• 设为首页
  • 点击收藏
  • 手机版
    手机扫一扫访问
    迪恩网络手机版
  • 关注官方公众号
    微信扫一扫关注
    公众号

AdelKS/LinuxGamingGuide: Linux Gaming Guide - An incomplete compilation of thing ...

原作者: [db:作者] 来自: 网络 收藏 邀请

开源软件名称(OpenSource Name):

AdelKS/LinuxGamingGuide

开源软件地址(OpenSource Url):

https://github.com/AdelKS/LinuxGamingGuide

开源编程语言(OpenSource Language):

Shell 100.0%

开源软件介绍(OpenSource Introduction):

A linux gaming guide

This is some kind of guide/compilation of things, that I got to do/learn about while on my journey of gaming on linux. I am putting it here so it can be useful to others! If you want to see something added here, or to correct something where I am wrong, you are welcome to open an issue or a PR !

Table of Content

Linux distribution

I have seen many reddit posts asking which linux distributions is "best" for gaming. My thoughts on the matter is that, to get the best performance, one simply needs the latest updates. All linux distributions provide the sames packages and provide updates. Some provide them faster than others. So any distribution that updates its packages the soonest after upstream (aka the original developers), is good in my opinion. Some distributions can take longer, sometimes 6 months after, for big projects (which is acceptable too, since one would get the updates without the initial bugs).

Lutris

Lutris is some kind of open source Steam that helps with installing and running some games. Each game has its own install script, maintained by usually different people (as far as I understand). I have only used Lutris, to install and run Overwatch, I don't think there's room for improvement in here since Lutris is just here to run overwatch with a chosen Wine version and environment variables. Correct me if I am wrong.

Some useful settings:

  • Enable FSYNC (if you have a patched custom kernel, further information below) otherwise enable ESYNC: once overwatch is installed, go to "Configure" > "Runner Options" > Toggle FSYNC or ESYNC.

Self-compiling

Compiling is the process of transforming human written code (like C/C++/Rust/... etc) to machine runnable programs (the .exe files on Windows, on Linux they usually have no extension :P). Compiling is actually done by a program, a compiler, on linux it's gcc or clang. There is not a unique way to translate/compile code to machine runnable programs, the compiler has lots of freedom on how to implement that, and we can influence them by telling them to try "harder" to optimize the machine code, by giving them the so called "flags": a set of command line options given to the compiler, an example is

gcc main.c -O2 -march=native -pipe

where -O2, -march=native and -pipe are compiler flags. There are many flags that compilers accept, the ones specific to optimization are given in GCC's documentation. A few important (meta)flags

  • The -Ox, where x=1,2,3, is a generic flag that sets the generic level optimization, it activates many other flags that actually do something. Distros compile the packages they ship usually with -O2
  • The -march flag is a flag that tells the compiler to use additional features that aren't available for all CPUs: newer CPU implement some "instruction sets" (aka additional features) that enable them to perform some tasks faster, like SIMD instructions. It makes some programs faster, like ffmpeg with video conversion. These instruction sets are not used by default in packages shipped by distros as they need to have them able to run on all machines, even those from 2001. So one can win some performance by just compiling with -march=native their computational heavy programs. Although some have embedded detection code to use additional instruction sets if detected. Some Linux Distributions like Gentoo enable you to compile every single package on your own machine so you can have ALL the apps built with -march=native (it may take several hours depending on your CPU)
  • Link Time Optimizations (LTO) that involve the use of the flags -flto, -fdevirtualize-at-ltrans and -flto-partition
  • Profile Guided Optimizations (PGO) that involve the use of -fprofile-generate=/path/to/stats/folder, -fprofile-use=/path/to/stats/folder flags. The idea behind is to produce a first version of the program, with performance counters added in with the -fprofile-generate=/path/to/stats/folder flag. Then you use the compiled program in your real life use-cases (it will be way slower than usual), the program meanwhile fills up some extra files with useful statistics in /path/to/stats/folder. Then you compile again your program with the -fprofile-use=/path/to/stats/folder flag with the folder /path/to/stats/folder filed with statistics files that have the .gcda extension.

A nice introduction to compiler optimizations -Ox, LTO and PGO, is made in a Suse Documentation that you can find here: https://documentation.suse.com/sbp/all/html/SBP-GCC-10/index.html

The Kernel, Wine, RADV and DXVK can be compiled on your own machine so you can use additional compile flags (up to a certain level) for the particular CPU you own and potentially faster with more "aggressive" compiler flags. I said potentially as you need to check for yourself if it is truly the case or not.

Flags to try

Here is a group of flags can use when building your own programs

BASE="-march=native -O3 -pipe"
GRAPHITE="-fgraphite-identity -floop-strip-mine"
MISC="-floop-nest-optimize -fno-semantic-interposition -fipa-pta"
LTO3="-flto -fdevirtualize-at-ltrans -flto-partition=one"
LTO2="-flto -fdevirtualize-at-ltrans -flto-partition=balanced"
LTO1="-flto -fdevirtualize-at-ltrans -flto-partition=1to1"

It is recommended to try them in the following order, if one fails (for whatever reasons: fails to compile or doesn't work), try the next one:

  1. BASE + GRAPHITE + MISC + LTO3:
-march=native -O3 -pipe -fgraphite-identity -floop-strip-mine -floop-nest-optimize -fno-semantic-interposition -fipa-pta -flto -fdevirtualize-at-ltrans -flto-partition=one
  1. BASE + GRAPHITE + MISC + LTO2:
-march=native -O3 -pipe -fgraphite-identity -floop-strip-mine -floop-nest-optimize -fno-semantic-interposition -fipa-pta -flto -fdevirtualize-at-ltrans -flto-partition=balanced
  1. BASE + GRAPHITE + MISC + LTO1:
-march=native -O3 -pipe -fgraphite-identity -floop-strip-mine -floop-nest-optimize -fno-semantic-interposition -fipa-pta -flto -fdevirtualize-at-ltrans -flto-partition=1to1
  1. BASE + GRAPHITE + MISC
-march=native -O3 -pipe -fgraphite-identity -floop-strip-mine -floop-nest-optimize -fno-semantic-interposition -fipa-pta
  1. BASE + GRAPHITE
-march=native -O3 -pipe -fgraphite-identity -floop-strip-mine
  1. BASE
-march=native -O3 -pipe

DXVK

This is the library that maps DirectX (Windows) to Vulkan (Multi-platform and open source) so games that are meant for Windows work on Linux. It's better than wine's built-in mapper called WineD3D. Lutris provides a version already.

You can compile your own latest one with some "better" compiler optimizations if you wish, and that's what I am doing but I have no idea about the possible FPS benefits of doing that. To do so you will need to put what DXVK's compile script gives you in ~/.local/share/lutris/runtime/dxvk/. Link here: https://github.com/doitsujin/dxvk

git clone https://github.com/doitsujin/dxvk.git
cd dxvk
# Build new DLLS
./package-release.sh master ~/.local/share/lutris/runtime/dxvk/ --no-package

Custom compile flags

DXVK can be compiled with user provided compile flags. For that, you edit build-win32.txt and build-win64.txt and change the following before running the ./package-release.sh script:

[built-in options]
c_args=[... TO BE FILLED ...]
cpp_args=[... TO BE FILLED ...]
c_link_args = ['-static', '-static-libgcc', ... TO BE FILLED ...]
cpp_link_args = ['-static', '-static-libgcc', '-static-libstdc++', ... TO BE FILLED ...]

Where you can replace ... TO BE FILLED ... with BASE + GRAPHITE + MISC + LTO3 flags defined here if you don't enable PGO. If you want to use PGO, you can use the BASE + GRAPHITE + MISC + LTO2 + -fprofile-generate=/path/to/dxvk-pgo-data or -fprofile-use=/path/to/dxvk-pgo-data, depending on the stage you are in. You can change the =/path/to/dxvk-pgo-data path. You also need to add '-lgcov' to c_link_args and cpp_link_args

Note: you need to respect the syntax of the build-winXX.txt files. Flags are quoted and separated with comas e.g. c_args=['-O2', '-march=native'].

These flag changes may improve performance or not, the best is to test with and without and see for oneself. If regressions happen or it doesn't want to compile you can try other flags.

GPU

  1. Update to the latest possible driver for your GPU
  2. If you are hesitating between AMD and Nvidia for your next GPU buy. As far as Linux is concerned: AMD all the way, because they are way more supported since they give out an open source driver.

Nvidia

The least one can do is redirect to Arch's documentation about it: https://wiki.archlinux.org/index.php/NVIDIA

If you didn't install the proprietary driver your computer is likely to be running an open source driver called nouveau, but you wouldn't want that to play games with that as it works based off reverse engineering and doesn't offer much performance.

Once you have the proprietary driver installed, open nvidia-settings, make sure you have set your main monitor to its maximum refresh rate and have 'Force Full Composition Pipeline' disabled (advanced settings).

Also, in Lutris, you can disable the size limit of the NVidia shader cache by adding __GL_SHADER_DISK_CACHE_SKIP_CLEANUP=1 to the environnement variables.

AMD

A nice documentation is given by, once again, Arch's documentation: https://wiki.archlinux.org/index.php/AMDGPU

  • "Very old" GPUs: the opensource driver is radeon and you only have that as an option, along with AMD's closed source driver I believe. But you are out of luck for running DXVK, since both driver's don't implement Vulkan.
  • "Old" GPUs: GCN1 and GCN2 are now supported by the newer "amdgpu" driver and you switch to it to win a few frames.
  • New GPUs: the base driver is amdgpu, and is shipped and updated with the linux Kernel, stacks on top of it three different drivers:
    • Mesa: the open source graphics stack that handles AMD, Intel, Qualcomm ...etc GPUs. The AMD OpenGL driver is called RadeonSI Gallium3D and is the best you can get. The Vulkan driver is called RADV
    • amdvlk: AMD's official open source Vulkan-only driver, I suppose the rest (OpenGL) is left to mesa. link here: https://github.com/GPUOpen-Drivers/AMDVLK
    • amdgpu PRO: AMD's official closed source driver, that has its own Vulkan and OpenGL implementation.

RADV

If you are running RADV and with a mesa version prior to 20.2, you should consider trying out ACO as it makes shader compilation (which happens on the CPU) way faster : go to "Configure" > "System Options" > Toggle ACO.

Your distro ships the latest stable version, you can go more bleeding edge to get the latest additions, but keep in mind that regressions often come with it. On Ubuntu there's a PPA that gives out the latest mesa, and another PPA that's less bleeding edge/more stable .

Self-compile

You can compile only RADV by hand with the extra bonus of using your own compiler optimizations as described in this section and use it for any Vulkan game, in a per game basis.

First, you get the source code

git clone --depth=1 https://gitlab.freedesktop.org/mesa/mesa.git

This command will create a mesa folder. To compile only RADV, you go into the sources folder and do the following

cd path/to/mesa
git clean -fdx
mkdir build && cd build
export CFLAGS="... [To be Filled] ..."
export CXXFLAGS="${CFLAGS}"
export LDFLAGS="-Wl,-O1,--sort-common,--as-needed,-z,now ${CFLAGS}"
meson .. \
    -D prefix="$HOME/radv-master" \
    --libdir="$HOME/radv-master/lib" \
    -D b_ndebug=true \
    -D b_lto=TO BE CHANGED \
    -D b_pgo=TO BE CHANGED \
    -D buildtype=release \
    -D platforms=x11,wayland \
    -D dri-drivers= \
    -D gallium-drivers= \
    -D vulkan-drivers=amd \
    -D gles1=disabled \
    -D gles2=disabled \
    -D opengl=false
meson configure
ninja install

Where you need to fill a a few lines

  • CFLAGS with flags, you can use BASE + GRAPHITE + MISC + LTO3 from the flags to try section.
  • If you enabled the LTO flags you must set -D b_lto=true, otherwise -D b_lto=false
  • With regards to PGO, first read the bullet point about PGO
    1. Profile generation
      • you must set -D b_pgo=generate
      • append -fprofile-generate=$HOME/radv-pgo-data to CFLAGS, where you can replace $HOME/radv-pgo-data by another folder if you wish
    2. Profile use
      • you must set -D b_pgo=use
      • append -fprofile-use=$HOME/radv-pgo-data to CFLAGS, where you replace $HOME/radv-pgo-data with the same folder you used for profile generation
    3. No PGO, you must set -D b_pgo=off

These may improve performance or not, the best is to test with and without and see for oneself. If regressions happen, follow the steps in flags to try section to reduce the number of flags.

After running the lines above, you get the driver installed in $HOME/radv-master, you can change the folder name and where it is in the line -D prefix="$HOME/radv-master". Now, to use it for Overwatch (or any other game), you must set the following environment variable (in Lutris, it's in "Configure" > "System Options" > Environment variables, and add it):

VK_ICD_FILENAMES=$HOME/radv-master/share/vulkan/icd.d/radeon_icd.x86_64.json:$OTHER_PATH/radeon_icd.i686.json

where you should manually replace $HOME by your home path /home/Joe and $OTHER_PATH by where radeon_icd.i686.json actually is, you can find out with

sudo updatedb
locate radeon_icd.i686.json

If the games crashes after doing all this, you can either try other git commits (you will need some git knowledge) or revert to the stable driver by simply removing the VK_ICD_FILENAMES environment variable. And if you don't wanna hear about bleeding edge mesa anymore you can simply remove the mesa source folder along with $HOME/radv-master.

Kernel

First, try to get the latest kernel your distro ships, it often comes with performance improvements (it contains the base updates for the amd gpu driver for example).

Command line options

As you may know, the kernel has various protection mechanisms from malicious program-execution based attacks, the likes of Spectre and Meltdown. These protections/mitigations come with an extra overhead on the CPU. (Un)fortunately, it is possible to disable ALL these mitigations, at the expense of security. Although if you use X11 then you are just adding an extra cherry on top of how unsecure your setup is haha. Since any running application can catch your keyboard and what's displayed on your monitor.

Okay, how to disable all mitigations ? Fortunately it's super simple: add mitigations=off command line to your kernel parameters.

I ran across another protection that got added in the kernel that disables a certain set of cpu instructions from user space programs (umip), instructions from this set is used for example in Overwatch some other games. That protection broke those games then the kernel got a patch that emulates those instructions with a certain overhead with a kernel message like this one:

kernel: umip: Overwatch.exe[5970] ip:140621a9a sp:21dea0: For now, expensive software emulation returns the result.

You can disable this protection with the following kernel parameter clearcpuid=514

Custom/self compiled kernels

Using a self-compiled kernel can bring some improvements. There is a git repository called linux-tkg that provides a script to compile the linux Kernel from source (takes about ~30mins, but can be stripped down with modprobed-db) with some customization options : the default scheduler (CFS) can be changed to other ones (Project C UPDS, PDS, BMQ, MuQSS) These changes help getting better performance in games. And also other patches. Linux-tkg needs to be compiled on your own machine, where you can use compiler optimizations such as -O3 and -march=native (LTO is experimental and Clang only, PGO will come soon), with an interactive script and a config file, I worked on the script to install on many distros other than Arch

More information here: https://github.com/Frogging-Family/linux-tkg

Threading synchronization

linux-tkg offers patches that makes the kernel better mimic Windows' behavior with threads. And therefore get better performance for games meant to run on Windows: winesync/fastsync, futex2, fsync, esync.

esync, fsync and futex2 names you may have heard about have been developed by CodeWeavers and Collabora. Chronologically, here's what happened

  • esync is the oldest implementation and available in any non ancient kernel in any distro, since it uses the kernel's eventfd system call. Issues arise in some distros when a game opens a lot of the so called "file descriptors"
  • FUTEX_WAIT_MULTIPLE, a special additional flag on the futex system call was developed and originally called fsync (that we will also call fsync1). linux-tkg and wine-tkg were offering support for this work under the fsync naming. This work did not get upstreamed in the linux kernel (out-of-tree).
  • futex2 implementing a new system call
    • Initially called futex_wait: linux-tkg and wine-tkg were offering support for this work under the futex2 naming.
    • The system call then got upstreamed in kernel 5.16 with a slightly different name : futex_waitv. Since then, it is referred to as fsync (so basically an fsync2) or futex2 interchangeably. Which leads to some confusions...

linux-tkg offers, through its customization.cfg file :

  • In kernels 5.16+
    • builtin support for futex_waitv
    • Support for fsync1 through futex_waitv to support fsync1 with old wines.
  • In kernels 5.15, 5.14 and 5.13
    • Back ported patches of futex_waitv thanks to these efforts (the original author behind the upstreaming effort).
  • For kernels < 5.16 : Support for "old" futex2 and fsync1. Old futex2 implementation exposed sysfs handles as its syscall number wasn't decided yet. On such kernels the following should output futex2: ls /sys/kernel | grep futex

winesync/fastsync is a new proposal of synchronization subsystem, similar to futex and eventfd, aimed to serve exclusively for mapping Windows API sync mechanisms. developed by wine developers. This implementation is put on hold since the upstreaming of futex_waitv. winesync is a kernel module that communicates with fastsync that should be in a patched wine (like wine-tkg). The performance should be similar or better than esync, but probably not better than fsync. To have the winesync module:

  • The DKMS route
    • Archlinux: you need to install the following package from the AUR: winesync, winesync, winesync-header and winesync-udev-rule
    • other distros: follow the README in this repository
  • Not offered by linux-tkg any longer

For a less efforts solution, you can look up Xanmod kernel, Liquorix, Linux-zen, Chaotic-AUR (Archlinux). That provide precompiled binaries. (futex2 is afaik not available in them).

Game mode

It's a small program that puts your computer in performance mode: as far as I know it puts the frequency scaling algorithm to performance and changes the scheduling priority of the game. It's available in most distro's repositories and I believe it helps in giving consistent FPS. Lutris uses it automatically if it's detected, otherwise you need to go, for any game in Lutris, to "Configure" > "System Options" > "Environment variables" and add LD_PRELOAD="$GAMEMODE_PATH/libgamemodeauto.so.0" where you should replace $GAMEMODE_PATH with the actual path (you can do a locate libgamemodeauto.so.0 on your terminal to find it). Link here: https://github.com/FeralInteractive/gamemode.

You can check whether or not gamemode is running with the command gamemoded -s. For GNOME users, there's a status indicator shell extension that show a notification and a tray icon when gamemode is running: https://extensions.gnome.org/extension/1852/gamemode/

AMD Ryzen: the cpuset trick

A small intro to CPU cache

The cache is the closest memory to the CPU, and data from RAM needs to go through the cache first before being processed by the CPU. The CPU doesn't read from RAM directly. This cache memory is very small (at maximum few hundred megabytes as of current CPUs) and this leads to some wait time in the CPU: when some data needs to be processed but isn't already in cache (a "cache miss"), it needs to be loaded from RAM. When the cache is "full", because it will always be, some "old" data in cache is synced back in RAM then replaced by some other data from RAM: this takes time.

There is usually 3 levels of cache memory in our CPUs: L1, L2, and L3. In Ryzen, the L1 and L2 are few hundred kilobytes and the L3 a (few) dozen megabytes. Each core has its own L1 and L2 cache, the L3 is shared: in zen/zen+/zen2 it is shared among each 4 cores (called a CCX). and CCX'es are grouped two by two in what is called CCDs. In zen 3, the L3 cache is shared among the cores of an entire CCD, 8 cores. There's this anandtech article that gives a through analysis of cache topology in Zen 2 vs Zen 3:

Zen3_vs_Zen2

One can obtain the cache topology if his current machine by running the following command:

$ lstopo

The lstopo of my previous Ryzen 3700X gives this

Ryzen 3700X topology

For my Ryzen 5950X gives this

Ryzen 5950X topology

What can we do with this information ?

Something really nice: give an entire CCX (for Zen/Zen+/Zen2 for CPUs that have six or more cores) or CCD (for Zen3, can only work with a 5950X or a 5900X) to your game, and make (nearly) everything else run in the other CCX(s)/CCD(s). With this, as far as I can hypothesize, one reduces the amount of L3 cache misses for the game, since it doesn't share it with no other app. The really nice thing with this, that you can notice easily, is that if you run a heavy linux kernel compilation on the other CCX(s)/CCD(s) your game is less affected: you can test for yourself. I think that using this trick also downplays the role a scheduler has on your games, since the game is alone and very few other things run with it on the same cores (like wine and the kernel).

Using cpuset

cpuset is a linux mechanism to create groups of cores (a cpu set) to which you can assign processes, at runtime. One can use it to create two cpu sets: one for your game, another for all the rest. Have a read at the doc to understand how things work. I may update things here to explain further.

A Ryzen CPU withtwo or more CCDs/CCXs can be split into two sets of cores, lets call the first one theGood (for games) and the second theUgly (for the rest). I made two similar scripts that do this, one for the Ryzen 3700X, that has 8 cores, 16 threads: logical cores 0-3,8-11 (given in the P# in the lstopo result) are assigned to theGood, which are associated to 4 physical cores (with SMT) in CCX0. Cores 4-7,12-15 are assigned to theUgly in CCX1 ; another one for for the Ryzen 5950X, that has 16 cores, 32 threads: logical cores 0-7,16-23 (given in the P# in the lstopo result) are assigned to theGood, which are associated to 8 physical cores (with SMT) in CCD0. Cores 8-15,24-31 are assigned to theUgly in CCD1. Then, the script redirects lutris to the theGood cpuset. Any process in a given cpu set will spawn child processes in the same cpu set, so lutris will launch wine and the game in the same cpuset automatically. You can edit the script to fit with your current CPU, after having a look at what lstopo outputs and at the cpuset documentation. You can reverse that cpu set creation and go back to no splitting between cores : created cpu sets (that are folders) can be removed if all the processes they contain get redirected to the main cpu set, that contains all cores. I made that script too.

important: core IDs should be carefully chosen so the cpu sets are separated by CCX/CCD and not just make a non hardware aware split (a recent AMD BIOS update changed the core naming scheme to fit with what Intel does), one way to verify it is, after doing the splitting, to call lstopo in both cpusets and verify. A way to do so is to move one shell to the new group, as root:

/bin/echo $$ >> /dev/cpuset/theGood/tasks
lstopo

Ryzen 3700X topology

Then also open another shell, and do lstopo, you should get separate results:

Ryzen 3700X topology

Benchmark

I did this benchmark on Overwatch, the conclusions are the following:

  • After a fresh restart, I already have a small number of processes (around 300), and most of them are sleeping, which means that Overwatch basically already has the entirety of the CPU for itself. Doing the cpuset trick reduced the performance: I think it's because Overwatch works optimally in more than 4 cores.
  • Playing while doing another heavy workload, like stream with software encoding, works better with the cpuset trick.

Wine

Wine is a program that enables running windows executables on Linux. Through Wine, windows executables run natively on your linux machine (Wine Is Not an Emulator xD), Wine will be there to remap all Windows specific behavior of the program to something Linux can handle, DXVK for example replaces the part of Wine that maps DirectX (Windows specific) calls of executables to Vulkan calls (That Linux can handle). Tweaking Wine can have quite the impact on games, both positive and negative. Latest wine from Lutris works fine, but wine-tkg in my experience performs better.

Environment variables

Some wine environment variables can be set that can help with performance, given that they can break games, they can be added on a per-game basis as usual in Lutris. The variables are the following:

STAGING_SHARED_MEMORY=1
STAGING_WRITECOPY=1

Wine-tkg

wine-tkg is a set of scripts that clone and compile wine's source code, on your own machine, with extra patches that offer better performance and better game compatibility. One of the interesting offered extra features are additional threading synchronization primitives that work with the corresponding patched linux-tkg kernel. One can use Esync+Fsync+Futex2 or fastsync (with its corresponding kernel module winesync).

Esync-Fsync-Futex2

To enable the use of Esync + Fsync + Futex2, wine-tkg needs to be built with the corresponding features enabled. Then, to enable Esync+Fsync+Futex2, you need to set the following environment variables

WINEESYNC=1
WINEFSYNC=1
WINEFSYNC_FUTEX2=1

Note that you can also run with only Esync or Esync+Fatsync by setting the variables to 0 (to disable) or 1 (to enable) accordingly. To know that esync, esync+fsync or esync+fsync+futex2 is running. You can try running your game/launcher from the command line and you should see one of the following:

  • esync:
    [...]
    esync: up and running
    [...]
  • esync+fsync:
    [...]
    fsync: up and running
    [...]
  • esync+fsync+futex2:
    [...]
    futex2: up and running
    [...]

esync+fsync+futex2 should be the fastest. But once again, you can only try to make sure.

Fastsync

To be able to use fastsync with wine-tkg, you need to do the following, in this order

  1. Be running a winesync enabled linux-tkg kernel, more information in this section
  2. Disable the use of wine-staging, fsync an

鲜花

握手

雷人

路过

鸡蛋
该文章已有0人参与评论

请发表评论

全部评论

专题导读
热门推荐
阅读排行榜

扫描微信二维码

查看手机版网站

随时了解更新最新资讯

139-2527-9053

在线客服(服务时间 9:00~18:00)

在线QQ客服
地址:深圳市南山区西丽大学城创智工业园
电邮:jeky_zhao#qq.com
移动电话:139-2527-9053

Powered by 互联科技 X3.4© 2001-2213 极客世界.|Sitemap