I use it via xcrun:
$ xcrun swift -v -o test test.swift
Swift version 1.0 (swift-600.0.34.4.5)
Target: x86_64-apple-darwin14.0.0
/Applications/Xcode6-Beta.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/swift -frontend -c -primary-file test.swift -enable-objc-attr-requires-objc-module -target x86_64-apple-darwin14.0.0 -module-name test -sdk /Applications/Xcode6-Beta.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.10.sdk -color-diagnostics -o /var/folders/2p/rs8p19s957ggyxzntnj3tp_40000gn/T/test-bb5ff8.o
/usr/bin/ld /var/folders/2p/rs8p19s957ggyxzntnj3tp_40000gn/T/test-bb5ff8.o -force_load /Applications/Xcode6-Beta.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/arc/libarclite_macosx.a -syslibroot /Applications/Xcode6-Beta.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.10.sdk -lSystem -arch x86_64 -L /Applications/Xcode6-Beta.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/swift/macosx -rpath /Applications/Xcode6-Beta.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/swift/macosx -macosx_version_min 10.10.0 -no_objc_category_merging -o test

Note however the swift libraries are loaded (via @rpath) from /Applications/Xcode6-Beta/.../swift/macosx so the binary won't work on another system unless it's got the same version of Xcode installed in the same place. Unless you want to do lots of copying/install_name_tool calls to sort it out...
$ otool -l test |fgrep path
         name @rpath/libswiftAppKit.dylib (offset 24)
         name @rpath/libswiftCoreGraphics.dylib (offset 24)
         name @rpath/libswiftDarwin.dylib (offset 24)
         name @rpath/libswiftDispatch.dylib (offset 24)
         name @rpath/libswiftFoundation.dylib (offset 24)
         name @rpath/libswiftObjectiveC.dylib (offset 24)
         name @rpath/libswift_stdlib_core.dylib (offset 24)
         path /Applications/Xcode6-Beta.app/Contents/Developer/Toolchains/XcodeDefault.x



export PATH=/Applications/Xcode6-Beta.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin:$PATH
3. vim test.swift
println("Hello swift")
4. 
xcrun swift -v -o test test.swift
5 ./test