在线时间:8:00-16:00
迪恩网络APP
随时随地掌握行业动态
扫描二维码
关注迪恩网络微信公众号
开源软件名称:WindRiver-OpenSourceLabs/ci-scripts开源软件地址:https://github.com/WindRiver-OpenSourceLabs/ci-scripts开源编程语言:Python 53.9%开源软件介绍:CI ScriptsThis repository contains scripts and Docker Compose files that form the Wind River Linux Continuous Integration Prototype. IntroductionI have built two build systems at WindRiver, but they were very specific to the WRLinux workflow and internal infrastructure. This is my third build system and an attempt to make a generic build and CI system with features around building and testing Yocto. This repo contains the scripts to orchestrate all the components as well as all the Dockerfile used to build the images. The images are hosted on Docker Hub and are automated builds linked to the Dockerfiles in this repository. All the images with links to back to the Dockerfile used to build them can be found here: https://hub.docker.com/r/windriver/ Features
Getting StartedRequirements:make python3: >= 3.3 Docker CE: >= 17.03 https://docs.docker.com/engine/installation/ Docker Compose: >= 1.13.0 https://docs.docker.com/compose/install/ Docker Socket permissionsIn order for Jenkins Agent to start docker containers without opening the socket to every user on the machine, the host system needs to allow a process with uid 1000 rw access to /var/run/docker.sock. This can be done by adding uid 1000 to the docker group. Currently if the host docker group has guid 995 to 999, this will enable access. If the /var/run/docker.sock does not sufficient permissions, the swarm client image will attempt to enable world rw permissions on the socket before attempting the build. Single Host SetupTo start Jenkins Master and Agent on a single system using docker-compose run:
This will download the images from the Docker Cloud/Hub and start the images using docker-compose. The Jenkins web UI is accessible at https://localhost/jenkins. If attempting to access the web UI from a different machine, replace localhost with the name or IP of the server where the repository was cloned to. The Jenkins interface is behind an nginx reverse proxy which uses a self signed certificate to provide TLS. The browser will warn you that the web page is using a cert from an unknown CA and require you to grant a security exception. Multi-Host SetupThe CI prototype supports distributing builds onto multiple machines using Docker Swarm.
The machine where the Using Docker Swarm requires some manual setup on each machine that will be part of the cluster. Each node needs Docker 17.03+ installed. On the node that will be master run:
and note the provided command line with join token. On the worker nodes run the provided command which will have the following form:
Limitations:
For more information on managing a Docker Swarm, consult the Docker Swarm Documentation Scheduling JobsBy default the Jenkins Master does not have authentication enabled and jobs can be submitted from any machine. On the same or a different machine, clone this repository. To install the python-jenkins package locally and submit a job run:
This will install all the python libraries into .venv and contact the Jenkins Master and schedule a build on the Jenkins Agent. The is a comma separated list of names from the yaml file. As examples the combos-WRLINUX_9_BASE.yaml contains names such as qemuarm_glibc-small_Base and qemux86-64_glibc-core_Base. The combos-WRLINUX_9_BASE.yaml file is a generated list of valid combinations of qemu bsps and configuration options. At WindRiver we generate these yaml files using the machines and images listed in the LayerIndex. Scheduling Poky BuildsAn example config is provided to demonstrate building Poky locally:
To reuse the ubuntu1604_64 image, the poky build uses the WRLinux buildtools from Github. Toaster IntegrationAll builds enable toaster by default and the prototype uses Registrator and Consul to discover toaster instances. The toaster aggregator webapp provides an overview of running toaster instances and the current progress of each as well as links to the individual toaster instances. The toaster aggregator web UI is available at https://localhost/toaster_aggregator Post build operationsThe conventional way to add post build operations on Jenkins Pipeline projects is to add stages to the pipeline. Each pipeline stage would also require more build parameters be added to the job configuration. Each additional post build step would require significant changes and would not compose well. The post build operations can be run in a different container than the build to avoid having to add tools like rsync to the build container. This also allows the build to run without network access. To select a different post build image:
The prototype contains a generic post build step that does not require modifications to job config or Jenkinsfile. The post build scripts are located in the scripts directory and can be selected to run using the command line:
This would run the scripts/rsync.sh and scripts/cleanup.sh scripts after a successful build and scripts/send_email.sh, scripts/cleanup.sh and scripts/report.sh after a failed build. To pass parameters to the post build scripts use:
The prototype will take the parameters, split them and inject them into the postbuild environment. To add post build steps, add a script to the scripts directory and add the script name to the --post_success and/or --post_fail and add any required parameters to --postprocess_args. Developer BuildsAn ideal Continuous Integration workflow supports the testing of patches before they are committed to the "trunk" branches. Also known as pre-merge testing or pull request testing. The standard git workflow is to use topic branches and have the CI system run tests against the topic branch. Yocto projects contain a hierarchy of git repositories and there isn't a standardized way to create a project area. The wr-lx-setup 1 attempts to standardize creation of Yocto projects using the Layerindex as a source of available layers and locations. This simplifies and standardizes the setup of Yocto projects, but it adds the Layerindex as a component required by the CI workflow. The prototype supports creation of a temporary per build Layerindex and modifying the attributes of a layer in the temporary Layerindex. This enables a developer to create a topic branch on a layer git repository and run builds and tests using this branch. An example workflow:
The sequence of events is:
Current Limitations:
Rsync serverTo support collection of build artifact results from a build cluster, an rsync server has been integrated. The runs beside the reverse proxy service and accepts files without authentication. The postbuild image now runs in the same network as the rsync server and can use the postbuild rsync script to copy files to this server or any external server. The contents of the rsync server are available over HTTPS through the
reverse proxy at Using Jenkins Credential Store to access Git serverJenkins has an encrypted credential store which can manage credentials used to access the git server. Connect to Jenkins and select Credentials in the left menu. Then select System -> "Global (unrestricted)" -> Add Credentials. Use ID "git" which is the default used by jenkins_job_submit.py.
Scheduling Jobs when Jenkins requires AuthenticationThe jenkins_job_submit.py can submit jobs to the Jenkins master that
requires Authentication. If unauthenticated access fails,
jenkins_job_submit.py check for a local file containing the
authentication credentials. The default name of the local auth file is
jenkins_auth.txt at the root path of this CI script. To use a
different file name use the command line option
The windriver/jenkins-master image requires authentication and has a special mechanism to retrieve this information transparently. If authentication to a different Jenkins server fails, contact the manager of Jenkins server and put the valid authentication in local auth file to submit jobs.
Runtime testsRuntime tests are supported using LAVA, which is a server running outside of CI scripts. The LAVA server configuration is set in config files which locates in configs folder. To enable runtime tests, the following settings in the YAML config file under configs folder are required: post_build:
post_process_image: postbuild
postprocess_args:
RSYNC_SERVER: yow-lpdtest.wrs.com
RSYNC_DEST_DIR: builds/wrlinux10
HTTP_ROOT: http://yow-lpdtest.wrs.com/tftpboot
REPORT_SERVER: http://yow-lpdtest.wrs.com:9200
SMTPSERVER: prod-webmail.windriver.com
EMAIL: [email protected]
post_success:
- rsync
- cleanup
post_fail:
- rsync
- cleanup
- send_mail
- report
test_config:
test: [disable (default) or test_suite_name]
test_image: postbuild
test_args:
LAVA_SERVER: <lava_server_link>
LAVA_USER: <lava_username>
NFS_ROOT: /net/yow-lpdtest/var/lib/tftpboot
TEST_DEVICE: [simics (default) or hardware]
RETRY: 1 Test SuitesIn above YAML config file, test under test_config section is used to set which test suite will be used for runtime test. Supported Runtime Tests
Configurations for each test suite are set in test_configs.yaml under configs folder. Here is and example: - name: oeqa-default-test
prebuild_cmd_for_test:
- test_configure.py
build_options:
- INHERIT += "testexport"
- IMAGE_INSTALL_append += "python3-pkgutil"
- IMAGE_INSTALL_append += "python3-unittest"
- IMAGE_INSTALL_append += "python3-multiprocessing"
- TEST_TARGET_IP = "localhost"
- TEST_SERVER_IP = "localhost"
- TEST_SUITES = "ping ssh df connman syslog xorg scp vnc date pam perl python rpm ldd smart dmesg dash"
lava_test_repo: git://ala-lxgit.wrs.com/lpd-ops/lava-test.git
simics:
job_template: lava-test/jobs/templates/wrlinux-10/x86_simics_job_oeqa-default-test_template.yaml
timeout: 420
hardware:
job_template: lava-test/jobs/templates/wrlinux-10/x86_64_job_oeqa-default-test_template.yaml
timeout: 480 Modifying docker imagesThe CI prototype uses the following images:
To test image modifications rebuild the container locally and run:
TODO
ContributingContributions submitted must be signed off under the terms of the Linux Foundation Developer's Certificate of Origin version 1.1. Please refer to: https://developercertificate.org To submit a patch:
Docker ImagesThis repository contains only the Dockerfiles used to generate the images. The images are assembled and hosted by Docker on the Docker Cloud/Hub. The images contain Open Source software as distributed by the following projects.
Included Third Party ComponentsMIT License
LicenseMIT License Copyright (c) 2017 Wind River Systems Inc. Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. |
2023-10-27
2022-08-15
2022-08-17
2022-09-23
2022-08-13
请发表评论