1、The first step is download tw folders, one comes from 

https://nlopt.readthedocs.io/en/latest/NLopt_on_Windows/ (64 version)

Another 64-bit Windows DLLs, built with MinGW, which comes from 

http://ab-initio.mit.edu/wiki/index.php?title=NLopt_on_Windows&redirect=no

 2、 Then cd into the folder with a bin subdirectory that contains libnlopt.dll, 

         use the command below to generate libnlopt.exports (the command is done under Developer Command Prompt for VS2013 )

dumpbin /EXPORTS yourfile.dll > yourfile.exports

Then select all the defined functions and copy them to a new file and name it as libnlopt.def, in the its header, add two lines: 

LIBRARY libnlopt.dll
EXPORTS

Then, still under Developer Command Prompt for VS2013, excute  command:

lib /def:libnlopt.def /machine:x64

Finally, do the following steps

% chage directory 
cd('Directory of nlopt');
% add the files to the search directory
addpath(genpath('Directory of nlopt'));
% build, link to header and link to .lib file (no space between -I, -L, and -l) 
mex .\matlab\nlopt_optimize.c -IDirectory of nlopt -LDirectory of nlopt -llibnlopt

the mex steup succeed!