在线时间:8:00-16:00
迪恩网络APP
随时随地掌握行业动态
扫描二维码
关注迪恩网络微信公众号
开源软件名称:dataux开源软件地址:https://gitee.com/mirrors/dataux开源软件介绍:Sql Query Proxy to Elasticsearch, Mongo, Kubernetes, BigTable, etc.Unify disparate data sources and files into a single Federatedview of your data and query with SQL without copying into datawarehouse. Mysql compatible federated query engine to Elasticsearch, Mongo,Google Datastore, Cassandra, Google BigTable, Kubernetes, file-based sources.This query engine hosts a mysql protocol listener,which rewrites sql queries to native (elasticsearch, mongo, cassandra, kuberntes-rest-api, bigtable).It works by implementing a full relational algebra distributed execution engineto run sql queries and poly-fill missing featuresfrom underlying sources. So, a backend key-value storage such as cassandracan now have complete Most similar to prestodb but in Golang, and focused oneasy to add custom data sources as well as REST api sources. Storage Sources
Features
Status
Try it OutThese examples are:
# download files to local /tmpmkdir -p /tmp/baseballcd /tmp/baseballcurl -Ls http://seanlahman.com/files/database/baseballdatabank-2017.1.zip > bball.zipunzip bball.zipmv baseball*/core/*.csv .rm bball.ziprm -rf baseballdatabank-*# run a docker container locallydocker run -e "LOGGING=debug" --rm -it -p 4000:4000 \ -v /tmp/baseball:/tmp/baseball \ gcr.io/dataux-io/dataux:latest In another Console open Mysql: # connect to the docker container you just startedmysql -h 127.0.0.1 -P4000-- Now create a new SourceCREATE source baseball WITH { "type":"cloudstore", "schema":"baseball", "settings" : { "type": "localfs", "format": "csv", "path": "baseball/", "localpath": "/tmp" }};show databases;use baseball;show tables;describe appearancesselect count(*) from appearances;select * from appearances limit 10; Big Query Example# assuming you are running local, if you are instead in Google Cloud, or Google Container Engine# you don't need the credentials or volume mountdocker run -e "GOOGLE_APPLICATION_CREDENTIALS=/.config/gcloud/application_default_credentials.json" \ -e "LOGGING=debug" \ --rm -it \ -p 4000:4000 \ -v ~/.config/gcloud:/.config/gcloud \ gcr.io/dataux-io/dataux:latest# now that dataux is running use mysql-client to connectmysql -h 127.0.0.1 -P 4000 now run some queries -- add a bigquery datasourceCREATE source `datauxtest` WITH { "type":"bigquery", "schema":"bqsf_bikes", "table_aliases" : { "bikeshare_stations" : "bigquery-public-data:san_francisco.bikeshare_stations" }, "settings" : { "billing_project" : "your-google-cloud-project", "data_project" : "bigquery-public-data", "dataset" : "san_francisco" }};use bqsf_bikes;show tables;describe film_locations;select * from film_locations limit 10; Hacking For now, the goal is to allow this to be used for library, so the # run dep ensuredep ensure -v Related Projects, Database Proxies & Multi-Data QL
We use more and more databases, flatfiles, message queues, etc.For db's the primary reader/writer is fine but secondary readerssuch as investigating ad-hoc issues means we might be accessingand learning many different query languages. Credit to mixer, derived mysql connection pieces from it (which was forked from vitess). Inspiration/Other works
from Databus BuildingI plan on getting the # for just docker# ensure /vendor has correct versionsdep ensure -update # build binary./.build# build dockerdocker build -t gcr.io/dataux-io/dataux:v0.15.1 . |
请发表评论