在线时间:8:00-16:00
迪恩网络APP
随时随地掌握行业动态
扫描二维码
关注迪恩网络微信公众号
开源软件名称:MMLSpark开源软件地址:https://gitee.com/mirrors/MMLSpark开源软件介绍:Synapse Machine LearningSynapseML (previously MMLSpark) is an open source library to simplify the creation of scalable machine learning pipelines.SynapseML builds on Apache Spark and SparkML to enable new kinds ofmachine learning, analytics, and model deployment workflows.SynapseML adds many deep learning and data science tools to the Spark ecosystem,including seamless integration of Spark Machine Learning pipelines with the Open Neural Network Exchange(ONNX),LightGBM,The Cognitive Services,Vowpal Wabbit, andOpenCV. These tools enable powerful and highly-scalable predictive and analytical modelsfor a variety of datasources. SynapseML also brings new networking capabilities to the Spark Ecosystem. With the HTTP on Spark project, userscan embed any web service into their SparkML models.For production grade deployment, the Spark Serving project enables high throughput,sub-millisecond latency web services, backed by your Spark cluster. SynapseML requires Scala 2.12, Spark 3.2+, and Python 3.6+.See the API documentation forScala and forPySpark. Table of ContentsFeatures
Documentation and ExamplesFor quickstarts, documentation, demos, and examples please see our website. Setup and installationPythonTo try out SynapseML on a Python (or Conda) installation you can get Sparkinstalled via pip with import pysparkspark = pyspark.sql.SparkSession.builder.appName("MyApp") \ .config("spark.jars.packages", "com.microsoft.azure:synapseml_2.12:0.9.5") \ .getOrCreate()import synapse.ml SBTIf you are building a Spark application in Scala, add the following lines toyour libraryDependencies += "com.microsoft.azure" % "synapseml_2.12" % "0.9.5" Spark packageSynapseML can be conveniently installed on existing Spark clusters via the spark-shell --packages com.microsoft.azure:synapseml_2.12:0.9.5pyspark --packages com.microsoft.azure:synapseml_2.12:0.9.5spark-submit --packages com.microsoft.azure:synapseml_2.12:0.9.5 MyApp.jar This can be used in other Spark contexts too. For example, you can use SynapseMLin AZTK by adding it to the DatabricksTo install SynapseML on the Databrickscloud, create a new library from Mavencoordinatesin your workspace. For the coordinates use: Finally, ensure that your Spark cluster has at least Spark 3.2 and Scala 2.12. If you encounter Netty dependency issues please use DBR 10.1. You can use SynapseML in both your Scala and PySpark notebooks. To get started with our example notebooks import the following databricks archive:
Apache Livy and HDInsightTo install SynapseML from within a Jupyter notebook served by Apache Livy the following configure magic can be used. You will need to start a new session after this configure cell is executed. Excluding certain packages from the library may be necessary due to current issues with Livy 0.5. %%configure -f{ "name": "synapseml", "conf": { "spark.jars.packages": "com.microsoft.azure:synapseml_2.12:0.9.5", "spark.jars.excludes": "org.scala-lang:scala-reflect,org.apache.spark:spark-tags_2.12,org.scalactic:scalactic_2.12,org.scalatest:scalatest_2.12" }} In Azure Synapse, "spark.yarn.user.classpath.first" should be set to "true" to override the existing SynapseML packages.Note that Azure Synapse is currently on spark 3.1, hence this custom version should be used on Spark 3.1 clusters. %%configure -f{ "name": "synapseml", "conf": { "spark.jars.packages": "com.microsoft.azure:synapseml_2.12:0.9.5-13-d1b51517-SNAPSHOT", "spark.jars.repositories": "https://mmlspark.azureedge.net/maven", "spark.jars.excludes": "org.scala-lang:scala-reflect,org.apache.spark:spark-tags_2.12,org.scalactic:scalactic_2.12,org.scalatest:scalatest_2.12", "spark.yarn.user.classpath.first": "true" }} DockerThe easiest way to evaluate SynapseML is via our pre-built Docker container. Todo so, run the following command: docker run -it -p 8888:8888 -e ACCEPT_EULA=yes mcr.microsoft.com/mmlspark/release Navigate to http://localhost:8888/ in your web browser to run the samplenotebooks. See the documentation for more on Docker use.
GPU VM SetupSynapseML can be used to train deep learning models on GPU nodes from a Sparkapplication. See the instructions for setting up an Azure GPUVM. Building from sourceSynapseML has recently transitioned to a new build infrastructure.For detailed developer docs please see the Developer Readme If you are an existing synapsemldeveloper, you will need to reconfigure yourdevelopment setup. We now support platform independent development andbetter integrate with intellij and SBT.If you encounter issues please reach out to our support email! R (Beta)To try out SynapseML using the R autogenerated wrappers see ourinstructions. Note: This feature is still under developmentand some necessary custom wrappers may be missing. PapersLearn More
Contributing & feedbackThis project has adopted the Microsoft Open Source Code of Conduct. For moreinformation see the Code of Conduct FAQ or contact[email protected] with any additionalquestions or comments. See CONTRIBUTING.md for contribution guidelines. To give feedback and/or report an issue, open a GitHubIssue. Other relevant projectsApache®, Apache Spark, and Spark® are either registered trademarks ortrademarks of the Apache Software Foundation in the United States and/or othercountries. |
请发表评论