* Remove copy constructor to make flatbuffers struct trivially copyable + add tests.
* Add support non c++11 compliant compilers.
* Fix std::trivially_copyiable test for non-C++11 compliant compilers.
* Fix trivially_copyable not part of glibc < 5 even with c++11 switch enabled.
In some debug environments using vector[i] does bounds checking even
though the standard specifies that it should not. Using
*(vector.begin()) sidesteps this though, giving the same result without
the bounds checking.
The travis script was only failing if the last Android build fails.
This changes the script to report a failure if any of the Android
projects fail to build.
This CL also introduces the following changes to allow the generation of the
Go library for flatbuffers:
- add support for --gen-onefile for Go to simplify the build rule (mapping each
input .fbs to a single separate .go file)
- add a new --go-import flag to override the default import line (currently
github.com/google/flatbuffers/go)
- add new go_library in BUILD for flatbuffer (for files in flatbuffers/go)
(mirrored from cr/171126159)
Change-Id: I83e705a9a9d9544837af0baf9366ec37757799aa
- Allowed enums to be declared before use.
- Generalized parsing of `required`.
- Reworked escaping of namespaces.
- Escaping field names that are C++ keywords.
- Many smaller fixes.
Change-Id: Ie580de7b70dc208f676f4f71bb0d061808648b8d
Tested: on Linux.
* Fix issue #4389, if field is ommitted the return type is the same for python generator
* Fix issue #4389, bytes is returned when field is omitted
* Fix issue #4389, added generated python code after generator modification
* Add constant accessors to C++ unions
* Remove redundant const pointer return type
* Update generate_code.bat to reflect generate_code.sh
* Add updated generated files
* Remove extra space from generated code
* Update generated files
* Change directory back to tests after generating code
* flatbuffers: Move EndianSwap template to flatbuffers/base.h
Clang complains
call to function 'EndianSwap' that is neither visible in the template definition nor found by argument-dependent lookup
return EndianSwap(t);
This seems to be due to limitation of two-phase lookup of dependent names in template definitions
Its not being found using associated namespaces therefore
it has to be made visible at the template definition site as well
Signed-off-by: Khem Raj <raj.khem@gmail.com>
* use __builtin_bswap16 when building with clang
clang pretends to be gcc 4.2.0 and therefore the code does
not use __builtin_bswap16 but tries to synthesize it
Signed-off-by: Khem Raj <raj.khem@gmail.com>
* Fix DetachedBuffer move assignment lifetime
DetachedBuffer move assignment now destroys it's own content,
rather than waiting for the destructor of other.
See more under #4435
Add missing function return type
* Rename empty_assign to reset
Reset after destroy
reset() now operates on itself
A Namespace object is now guaranteed unique.
This cleaned up some old workarounds and latent bugs.
Change-Id: Ic3f12d89947871b03b2c449ba51b3186f953adde
Tested: on Linux.
Bug: 21336857
This is something the format supports, but none of the builders
were doing. Can save 10-20% on FlatBuffer binary size!
Also fixed the Go tests.
Change-Id: I616c56ce9bbcfcaee23aa24f0532fcb60b6a8c75
Tested: on Linux.
* Eclipse ignore
* TypeScript support
* Prefixing enums
* Test results
* Merged JS and TS generators
* Fixed AppVeyor build problems
* Fixed more AppVeyor build problems
* Fixed more AppVeyor build problems
* Changed TS flag to options struct
* Storing options by value
* Removed unneeded const
* Re-export support for unions
* Uint support
* Casting bools to numbers for mutation
* TS shell tests
* Reverted generates js test file to original version
* Backing up js tests and properly generating test data
* Not importing flatbuffers for TS test generation
* Not overwriting generated js for tests
* AppVeyor test fixes
* Generating the most strict TS code possible
* Not returning null when creating vectors
* Not returning null from struct contructors
* Vector of unions for ts/js
* Sanity check for languages
* Indentation fix + output test files
* Vectors of unions for php
* Fixes to union vector handling + tests
* Eclipse ignore
* TypeScript support
* Prefixing enums
* Test results
* Merged JS and TS generators
* Fixed AppVeyor build problems
* Fixed more AppVeyor build problems
* Fixed more AppVeyor build problems
* Changed TS flag to options struct
* Storing options by value
* Removed unneeded const
* Re-export support for unions
* Uint support
* Casting bools to numbers for mutation
* TS shell tests
* Reverted generates js test file to original version
* Backing up js tests and properly generating test data
* Not importing flatbuffers for TS test generation
* Not overwriting generated js for tests
* AppVeyor test fixes
* Generating the most strict TS code possible
* Not returning null when creating vectors
* Not returning null from struct contructors
* Copying required field to union type vector
* Removed unneccessary bool
* Add support for boolean types in flexbuffers
* Simplify casting number <=> boolean
* Added comments for tests
* Add proper support for Booleans
* Bad rebase
* No special case for strings
* Removed unused test
* Simplify logic
* Add numpy accessor to python flatbuffers scalar vectors
* Update python tests to test numpy vector accessor
* Update appveyor CI to run Python tests, save generated code as artifact
* Update example generated python code
* Add numpy info to python usage docs
* Update test schema and python tests w/ multi-byte vector
* did not mean to push profiling code
* adding float64 numpy tests
* added bool for nested_flatbuffer
setting bool flag nested_flatbuffer according to attributes
setting nested type
* added JSON serialization for nested flatbuffers
* simplified lookup of nested_flatbuffer StructDef
* added nested_flatbuffer parsing workound relying on flexbuffers
* moved nested flatbuffer parsing into its own function for clarity
removed flexbuffers workound to simplify and speed up nested_flatbuffer parsing
added support for 'legacy' nested_flatbuffers, ubyte arrays
* inlined functions for nested parser init/clean since only used once
* whitespace formatting
* changed type of FieldDef.nested_flatbuffer from bool to StructDef*
removed subsequent type lookups
* removed copies of unrequired data when initializing nested parser
* applied changes requested by reviewer
* removed superfluous lookup of nested_flatbuffer field attributes
* renamed camelCased variables to snake_case
* Added internal - limited - implementation of flatbuffers::unique_ptr
for STLs that don't ship with std::unique_ptr. In C++11 and beyond
this is just an alias for std::unique_ptr.
* Aliased used type traits structs is_scalar is_floating_point is_unsigned
into flatbuffers namespace so they can be replaced in C++98 implementations.
Right now these point at stlport's TR1 implementations.
* Wrapped vector::data() in vector_data().
* Wrapped vector::emplace_back() in vector_emplace_back().
* Wrapper string::back() in string_back().
* Added variants of FlatBufferBuilder::CreateVector() and
FlatBufferBuilder::CreateVectorOfStructs() that allow the use of plain
function pointers.
Generated code has also been modified to use plain functions to build objects
rather than std::function() so all generated code will work in C++98
applications.
* Added flexbuffers::Builder::Vector(), flexbuffers::Builder::TypedVector()
and flexbuffers::Builder::Map() methods that allow the use of plain function
pointers.
* Changed Parser to internally use plain function pointers when parsing table
and vector delimiters.
* Added specializations of NumToString() for 64-bit types that aren't supported
by stringstream in stlport.
* Overloaded numeric_limits for 64-bit types not supported by stlport.
* Replaced build_apk.sh (which was broken by deprecation of the
"android" tool in the Android SDK) with build.gradle and the
appropriate gradle wrapper to build an APK.
* Switched Android build to build against all STL variants.
* Updated travis configuration to build Android test and sample.
Tested:
* Verified all tests continue to work on Linux, OSX and Android.
* Verified Travis build is green.
Change-Id: I9e634363793f85b9f141d21454b10686020a2065
* idl_parser.cpp ignores $schema in input json
#4381
* idl_parser.cpp: Fixed ci error signed unsigned comparison
* idl_parser.cpp: replaced case insensitive search for $schema with case sensitive search
* idl_parser.cpp: Expecting a string constant after $schema
#4382
The ByteBufferFactory interface gives the user an option to specify
the method in which the internal ByteBuffer is allocated. This provides
flexibility in the type of ByteBuffer that can be used.
The sizedInputStream method is an alternative to sizedByteArray that
does not make a copy of the data in memory.
* Added empty generator for json schema (idl_gen_json_schema.cpp)
#4360
* JsonSchemaGenerator: output of tables implemented
current problems:
- typenames are not correct
- array types need to be deduced
#4360
* JsonSchemaGenerator: Corrected generation of typenames
Current problems: enum types not written correctly
#4360
* JsonSchemaGenerator: Added generation of enum types
#4360
* idl_gen_json_schema.cpp: Write required properties to schema
#4360
* idl_gen_json_schema.cpp: Export Types including namespace
#4360
* idl_gen_json_schema.cpp: Fixed Json format
#4360
* idl_gen_json_schema.cpp: Formatted according to google code style
#4360
* Checked in monster_test.bfbs with changes from master
* Added idl_gen_json_schema.cpp in CMakeLists.txt
* generate_code.bat: Added generation of json schema
* Added todo.md
* generate_code.sh: Added generation of json schema
* Addressed some review issues
- removed command line arg -S
- removed new lines
- fixed codestyle in template functions
- removed usage of stringstream
- idented json schema
#4360
* removed auto in idl_gen_json_schema.cpp
* idl_gen_json_schema.cpp: changed iterator declarations to auto
#4360
* deleted todo.md
* idl_gen_json_schema.cpp: Removed keyword "override" so that vs2010 can compile
* idl_gen_json_schema.cpp: switch statement in GenType handeles all enum-members
* idl_gen_json_schema.cpp: Removed cerr output
* idl_gen_json_schema.cpp: Avoid vector copying
* idl_gen_json_schema.cpp: Fixed identation of json schema output
* idl_gen_json_schema.cpp: Do not output empty descriptions
flatbuffers.h(591): error C2220: warning treated as error - no 'object' file generated
flatbuffers.h(591): warning C4267: 'return': conversion from 'size_t' to 'flatbuffers::uoffset_t', possible loss of data
SerializationTraits<T>::Deserialize _transfers_ ownership of the buffer,
so we must destroy it.
This commit also includes some misc fixes:
- Use grpc::Status::OK rather than default ctor for clarity.
- Check for a null buffer passed into Deserialize, and handle it the
same way as the protobuf deserializer.
* Extended json parsing capability: add support for parsing nested lists and top level lists
* Stylistic conformance with surrounding code + generalized comments
* More code tidy-up for stylistic conformance with surrounding code
* Blank lines
* Reverted changes related to top-level list parsing
* Styling: newline before else
* Taking out ProcessTableFields which is no longer needed as the top level list change was reverted.
* [C++] Remove std::iterator usage (#4340)
Inheriting from std::iterator has never been required, and it's
deprecated in C++17. It can be replaced by directly providing typedefs.
Include <iterator> for std::random_access_iterator_tag.
Note that structs default to public access control.
* [C++] Change whitespace style in typedefs.
* Rework flatbuffers + gRPC integration
- Introduce `flatbuffers::grpc::Message<T>`, a `grpc_slice`-backed
message buffer that handles refcounting and allows flatbuffers to
transfer ownership to gRPC efficiently. This replaces
`flatbuffers::BufferRef<T>`, which required a copy call and was also
unsafe w.r.t. buffer lifetime.
- Introduce `flatbuffers::grpc::MessageBuilder`, a gRPC-specific builder
that forces a `grpc_slice`-backed allocator and also adds some helpful
`Message<T>`-related methods.
- Update serializers accordingly (now zero-copy between flatbuffers and
gRPC).
* gRPC: verify messages by default, but allow user to override
* gRPC: fix some formatting issues
* Disable verification by default, but add helper method
* Make FlatBufferBuilder fields protected + remove vec accessor
* Use bool add_ref parameter to toggle refcount incr
* Remove unnecessary inline specifiers
* Fix formatting
* Use auto
* Remove empty lines
* Use grpc_slice helper macros
* Simplify reset code
* Disable Message copy ctor and assignment by default
* Remove unused member
* Enable gRPC verification by default
* Use auto
* Bake in message verification (remove template specialization)
* Add RoundUp func
* Consolidate gRPC message copy flag
* Make vector_downward allocations fully lazy
* Test message verification failure code/message
* Add grpctest verification test comments
* Simplify reallocate implementation
* Make initial_size a size_t
* Use ternary op for growth_policy
* Use truthiness rather than dont explicit nullptr check
* Indent preprocessor directives
* Remove grpc message copy/assignment
* Fix a few bugs
* Add gRPC example
* Add basic gRPC docs
* Use doxygen EXAMPLE_PATH + @include
* Reference example fbs in grpc docs
* Move gRPC examples into grpc/samples
* Fix pointer/reference formatting
* Use std::function rather than templated callback func
* Create fresh message builder for each request
* Use Clear() in Reset() impl
* Use FLATBUFFERS_CONSTEXPR
* Remove std::move around a raw pointer
Calling std::move on a raw pointer has no advantage to just copying its value. Moreover, it is confusing, because it indicates that the argument is movable in some non-trivial way (e.g., is it actually meant to be a smart pointer?). More context in https://crbug.com/729393.
* Remove the move constructor altogether
FlexBuffers are actually completely unrelated to FlatBuffers and can be used separately.
However, since they utilize several utility functions from flatbuffer.h, they require this header file, which creates unreasonable dependency.
By moving those utility functions to separate base header, both libraries can use same code but still be independent
* Track included files in PATH-agnostic way.
Use full paths as keys in the map of included files. Store logical
include path as a value, in order to put it to the generated file.
* Fix tests by accepting null |include_filename|.
* Fix self-includes code generators.
Changing to keep include prefixes had two side effects: the main
file being parsed wasn't filtered out anymore, and include directory
paths would be added to the path in the include statement.
Also moved the include_test*.fbs files to sub directories so we
can actually test the handling of -I etc.
tested: on Linux.
Change-Id: Ibae095cea7ab0cccbac15cfb5171719f6b5cad8c
* Improve Allocator handling
- Templatize Allocator on vector_downward, and make it own the allocator
instance so it can manage lifetimes.
- Templatize + rename FlatBufferBuilderT accordingly, and add a typedef
to FlatBufferBuilder so old code continues to work.
- Fix some issues with the release deleter
- More details in github issue #4311
* Fix constexpr for older MSVC
* Reimplement allocator improvements via inheritance
Instead of templates, use an abstract base class and some unique_ptrs to
implement the new and improved allocator.
* Fix misplaced newline
* Add missing override keyword
* Add macro for func delete to support older compilers
* Explicitly move BufferDeleter (maybe fixes VS 10?)
* Revert previous attempt at VS10 fix
* Try yet another workaround for MS 10
* Use FLATBUFFERS_NOEXCEPT macro
* Add FLATBUFFERS_OVERRIDE macro
* Fix some issues with MSVC 16 hack
* Remove dep on unique_ptr
* Update DEPRECATED macro with a message
* Sigh, no delegating ctors before c++11
* Fix simple_allocator stub
* Relax FlatBufferBuilder ctor
* Add vector_downward reset + assert in clear
* Rename DetachedBuffer to FlatBuffer + remove unnecessary deprecations
* Add flatbuffers::FlatBufferBuilder::Release()
* Address remaining allocator-improvements comments
There were several possible bugs involving paths not being
recognized as being the same on Windows. Rather than trying
to ensure all code deals with / and \ correctly, paths now
get transformed to / on input, fixing all current and
future such bugs.
Tested: on OS X.
* Eclipse ignore
* TypeScript support
* Prefixing enums
* Test results
* Merged JS and TS generators
* Fixed AppVeyor build problems
* Fixed more AppVeyor build problems
* Fixed more AppVeyor build problems
* Changed TS flag to options struct
* Storing options by value
* Removed unneeded const
* Re-export support for unions
* Uint support
* Casting bools to numbers for mutation
* TS shell tests
* Reverted generates js test file to original version
* Backing up js tests and properly generating test data
* Not importing flatbuffers for TS test generation
* Not overwriting generated js for tests
* AppVeyor test fixes
* Generating the most strict TS code possible
* Not returning null when creating vectors
* Not returning null from struct contructors
* Added <autogenerated> in cs generator file header comment
#4287
* CS: Added xml-correct file header comment including "<auto-generated>...</auto-generated>"
code_generators.cpp: FlatBuffersGeneratedWarning() function replaced by compile time constant "GeneratedWarning"
also removed extra newlines at end of GeneratedWarning to be able to generate a xml well formed file comment for cs files
#4291
* code_generators: Changed static string GeneratedWarning back to function FlatBuffersGeneratedWarning()
#4291
* Added modified Unit-Test files
#4291
* idl_gen_general: Add autogenerated only in C# code
#4291
Since it wasn't documented and very different from a vector of
tables, this has caused a lot of confusion in the past.
Change-Id: Iab47c61b55c19abe5c4f25c86d71335a6b6321ca
Previously, we had a check to simply skip such files, but this
tends to make build systems unhappy.
This only affects C++ and JS, since other language output per-class
files.
Change-Id: I54224642725bbafb9f6e1654ed3693e62ca9f7d7
Tested: on Linux.
* Don't fail the build on unused parameters
gRPC headers have unused parameters so this breaks the test build.
* Pull in updated compiler files from gRPC
There have been some API breaks in gRPC lately. This commit
pulls in the most recent version of the files in this repo
that are just copied from gRPC.
* Modify the gRPC files so that they can work with Flatbuffers
The files taken from gRPC do not work out-of-the-box with Flatbuffers.
This commit modifies them so that they work. Hopefully this commit
will be able to serve as a guide or maybe even be cherry-picked on
top of new versions of those files as newer versions from gRPC are
pulled in.
* Adjust the rest of Flatbuffers to work with the new gRPC
* Change idl_gen_grpc.cpp to work with the new API
* Add missing #include in flatbuffers/grpc.h
* Run tests/generate_code.sh and check in the results
* Don't link with grpc++_unsecure and (secure) grpc. That's just weird
* Revert unrelated JS/TS test changes
* Simplify compiler/config.h
There is no need to import this file from gRPC. In fact, it probably
makes sense to not do so, since it seems to be intended to have
project specific configuration in it.
* Don't emit C++ types in the Go gRPC code generator
* Don't emit C++ #includes in generated gRPC code
Before this PR, there was a Go-specific additional_includes method
in schema_interface.h, which is shared with the gRPC repo. The
additional parameter to FlatBufFile in idl_gen_grpc.cpp makes that
unnecessary, which means we need less Flatbuffer-specific changes
in gRPC.
This is because it trips up the Closure compiler with errors like:
ERROR - assignment to property bb of SomeTable
found : undefined
required: (flatbuffers.ByteBuffer|null)
this.bb = undefined;
^^^^^^^^^^^^^^^^^^^
Change-Id: Iaf032b5249ec655e151331e81532e549c12bcd78
GCC gained a new warning, -Wimplicit-fallthrough, which warns about
implicitly falling through a case statement. The regular expressions
used at the default level (-Wimplicit-fallthrough=3) don't match with
a colon at the end. The comment also needs to be followed (after
optional whitespace and other comments) by a 'case' or 'default'
keyword, i.e. it will not be recognized with a '}' between the comment
and the keyword.
* Eclipse ignore
* TypeScript support
* Prefixing enums
* Test results
* Merged JS and TS generators
* Fixed AppVeyor build problems
* Fixed more AppVeyor build problems
* Fixed more AppVeyor build problems
* Changed TS flag to options struct
* Storing options by value
* Removed unneeded const
* Re-export support for unions
* Uint support
* Casting bools to numbers for mutation
* TS shell tests
* Reverted generates js test file to original version
* Backing up js tests and properly generating test data
* Not importing flatbuffers for TS test generation
* Not overwriting generated js for tests
* AppVeyor test fixes
* Generating the most strict TS code possible
Zero offsets are non-sensical in FlatBuffers (since offsets are
relative to themselves) but were allowed by the verifier. This could
cause buffers made up of all zeroes to be interpreted as correct
buffers with an empty root object.
Generally, not allowing such offsets will make the verifier more
likely to catch problems earlier.
Change-Id: I54010bea29721b326ff8e5348fcd9fe78e5e7506
Tested: on Linux.
* Eclipse ignore
* TypeScript support
* Prefixing enums
* Test results
* Merged JS and TS generators
* Fixed AppVeyor build problems
* Fixed more AppVeyor build problems
* Fixed more AppVeyor build problems
* Changed TS flag to options struct
* Storing options by value
* Removed unneeded const
* Re-export support for unions
* Uint support
* Casting bools to numbers for mutation
* TS shell tests
* Reverted generates js test file to original version
* Backing up js tests and properly generating test data
* Not importing flatbuffers for TS test generation
* Not overwriting generated js for tests
* AppVeyor test fixes
* Added support for serializing native_type with CreateVectorOfNativeStructs
* Added support for serializing native_type with CreateVectorOfSortedNativeStructs
* Added C++ code generation for vectors of native_types
Move constructors are present, which it should use instead.
This is a temp fix to make it compile, but eventually we should
generate a proper copy constructor just in-case people want to
copy objects with unions.
Tested on: Linux, OS X.
Change-Id: Idf85419995c96f5959061882157541573e306083
* fix bounds checking on integer parsing
the previous code was allowing 255 for int8_t, similar for int16_t
and int32_t, and even negative values for unsignd types.
this patch fixes bounds checking for 8-bit, 16-bit and 32-bit types.
testing for both acceptable values and unacceptable values at the
boundaries are also improved.
bounds checking on 64-bit types isn't addressed by this patch.
* fix 'unary minus operator applied to unsigned type, result still unsigned'
* fix & placement
(C++ only for now).
Also fixed vector of union support in the object API.
Bug: 36902939
Change-Id: I935f4cc2c303a4728e26c7916a8ec0adcd6f84cb
Tested: on Linux.
* Fix Visual Studio 2017 new warning (C4244: 'argument': conversion from 'int' to 'const char', possible loss of data)
* Fix Visual Studio 2017 pedantic warnings
* Fix Visual Studio 2017 pedantic warnings
* Eclipse ignore
* TypeScript support
* Prefixing enums
* Test results
* Merged JS and TS generators
* Fixed AppVeyor build problems
* Fixed more AppVeyor build problems
* Fixed more AppVeyor build problems
* Changed TS flag to options struct
* Storing options by value
* Removed unneeded const
* Re-export support for unions
* Uint support
* Casting bools to numbers for mutation
* TS shell tests
* Reverted generates js test file to original version
* Backing up js tests and properly generating test data
* Use noexcept in union type move ctor/Add move assingment
* Add NOEXCEPT macro to deal with _MS_VER/Remove delegating ctor in union type class
* Add FLATBUFFERS_NOEXCEPT to generated union class
* C#: Fixed possible conflicts between usings and user-supplied namespace
C#: Added the global qualifier to using directives to prevent possible conflicts with the user-supplied namespace. Also prevents unintentional type hiding. Resolves issue #4242.
* Updated C# generated code files
* Support binary search for struct in cpp
CreateVectorOfSortedStruct is provided for convenience.
* fix continuous-integration error
* add generated files
* compile Ability.cs in csharp test
* compile Ability.cs in csharp
* modify according to code review
Vector<Offset<T>> to Vector<Offset<U>> if U is a base
class of T.
This is useful for when you want to generically
iterate over a vector of objects that derive from
flatbuffers::Table.
Change-Id: I59161e3b9f40501f72e02b46509be9dc8ab86c6b
Building on Arduino fixed by conditional includes of cstdint and utility. In the Standard C++ for Arduino (port of uClibc++):
* cstdint is not present.
* utility is named utility.h.
Replaced size_t with uoffset_t for Verifier::max_tables_, max_depth_, depth_, and num_tables_ to ensure 32-bit values are used (and not 16-bit) - gave rise to a warning.
* Add default value handling to mutation/SetField code
* Shorten reflection SetField impl
* Modify impl to work with C++03
* Add more mutation tests
* Fail SetField if non-scalar
* Add IsScalar/IsInteger/IsFloat for reflection::BaseType
* Use new IsScalar/IsInteger/IsFloat in reflection SetField
* Assume scalar is either int or float
* Pedantic Visual Studio build (/W4 /WX)
* Pedantic Visual Studio build (/W4 /WX)
* Pedantic Visual Studio build (/W4 /WX)
* Remove /wd4512 from CMakeLists.txt
* Suppress Visual Studio 4512 warning ('class' : assignment operator could not be generated)
optimization of FlatBufferBuilder::CreateVector()
1. optimization of FlatBufferBuilder::CreateVector() for "1 == sizeof(T)" ( such as [byte], [ubyte]).
2. For my project, it was about 10x improvement on flatbuffers serialization.
3. why not "string": "string, which may only hold UTF-8 or 7-bit ASCII. For other text encodings or general binary data use vectors ([byte] or [ubyte]) instead."
* Additional accessors for fields of union types.
* Reveal the hidden parameter.
* External explicit specializations for templated accessors.
* Changes according to code style. Generated code included as well.
* Update idl_gen_cpp.cpp
Fix for: error C2668: 'Service::Version2::CreateServiceValues' : ambiguous call to overloaded function #4145
* Update idl_gen_cpp.cpp
Also needed the same change in the getPostTable
* Add files via upload
New generation from running generate_test.bat
* Add files via upload
Lets try this without added line feeds
* Added a test for a potential "ambiguous call to overloaded function"
* Change to a better name of the table.
Didn't like the Service in the name.
* Reverted back to the original
It passed all scalar ints thru a int64_t, which would truncate
uint64_t values with the upper bit set.
Change-Id: I38fb8c68c911ae44d9863f8e35c2429ca0ab51e5
Tested: on Linux.
Allow tables to be mapped to native types directly. For example, a table
representing a vector3 (eg. table Vec3 { x:float; y:float; z:float; }) can
be mapped to a "mathfu::vec3" native type in NativeTables. This requires
users to provide Pack and UnPack functions that convert between the
Table and native types. This is done by adding the "native_type" attribute
to the table definition.
To support user-defined flatbuffers::Pack and flatbuffers::UnPack functions,
support a "native_include" markup that will generate a corresponding
Also add an UnPackTo function which allows users to pass in a pointer to
a NativeTable object into which to UnPack the Table. The existing UnPack
function is now simply:
NativeTable* UnPack() {
NativeTable* obj = new NativeTable();
Table::UnPackTo(obj);
return obj;
}
Finally, allow native types to be given a default value as well which are
set in the NativeTable constructor. This is done by providing a
"native_default" attribute to the member of a table.
Change-Id: Ic45cb48b0e6d7cfa5734b24819e54aa96d847cfd
This would happen if they were supplied in an order that does not match
the schema relative to other fields. It now supports any ordering.
Change-Id: I9d309cd4e6e5c470f01d9d431806eba4f9f46559
Tested: on Linux.
The conditions to add the "NONE" or "ANY" value in the enum were useless
because the user cannot provide a zero value or a bitmask (for "ANY") in
the bit_flags enum type.
Helps simplify code generation code. Instead of this:
code += "inline const " + cpp_qualified_name + " *Get";
code += name;
code += "(const void *buf) {\n return flatbuffers::GetRoot<";
code += cpp_qualified_name + ">(buf);\n}\n\n";
You do this:
code.SetValue("NAME", struct_def.name);
code.SetValue("CPP_NAME", cpp_qualified_name);
code += "inline const {{CPP_NAME}} *Get{{NAME}}(const void *buf) {";
code += " return flatbuffers::GetRoot<{{CPP_NAME}}>(buf);";
code += "}";
code += "";
Updated the CPP code generator to use the CodeWriter class. Most of the
changes in the generated code are white-space changes, esp. around new
lines (since the code generator class automatically appends new lines
when appending a string). Actual code changes include:
* Renamed "rehasher" to "_rehasher" for consistency with other args in
Pack function.
* Renamed "union_obj" to "obj: in UnPack function.
* Always do "(void)_o;" to prevent unused variable warning in Create
function (instead of only doing it if there are no fields) in order
to avoid two-passes.
* Renamed padding variables from __paddingX to paddingX__.
"Each name that contains a double underscore (_ _) [...] is reserved
to the implementation for any use." C++ standards 17.4.3.1.2.
* Add braces around switch cases.
* Calculate index as a separate statement in EnumName function, eg.
const size_t index = ...;
return EnumNamesX()[index];
vs.
return EnumNamesX()[...];
* Stored end table offset in variable in Finish() functions, eg.
const auto end = fbb_.EndTable(start_, ...);
auto o = flatbuffers::Offset<T>(end);
vs.
auto o = flatbuffers::Offset<T>(fbb_.EndTable(start, ...));
* Separate reinterpret_cast calls from function calls in Union
functions, eg.
auto ptr = reinterpret_cast<const T *>(obj);
return ptr->UnPack(resolver);
vs.
return reinterpret_cast<const T *>(obj)->UnPack(resolver);
* Removed unecessary (void)(padding__X) no-ops from constructors, eg.
Test(int16_t a, int8_t b) : ... {
(void)__padding0; // <-- Removed this line.
}
In the idl_gen_cpp.cpp file itself, I refactored some code generation into
new functions: GenParam, GenNativeTable, GenVerifyCall, GenBuilders,
GenUnpackFieldStatement, and GenCreateParam.
Change-Id: I727b1bd8719d05b7ce33cbce00eb58fda817b25d
- Update to be const-correct where possible.
- Consistently pass |code| as pointer instead of non-const-ref.
- No newlines (\n) characters in the middle of code strings.
- Use if-else if-else statements instead of nested ternary operators.
- Ensure all lines end at 80 chars.
- Make utility functions static.
From cl/143505731.
Change-Id: If0fab9ee75de5af963367a948dddf53af93f73b4
3a1f776132 added a required identifier parameter
to Verifier::VerifyBuffer but did not update the templated call site in
BufferRef. This causes errors like the following when trying to instantiate the
call site by calling BufferRef::Verify():
include/flatbuffers/flatbuffers.h: In instantiation of ‘bool flatbuffers::BufferRef<T>::Verify() [with T = MyType]’:
MyApp.cpp:16:3: required from here
include/flatbuffers/flatbuffers.h:1421:37: error: no matching function for call to ‘flatbuffers::Verifier::VerifyBuffer()’
return verifier.VerifyBuffer<T>();
^
include/flatbuffers/flatbuffers.h:1421:37: note: candidate is:
include/flatbuffers/flatbuffers.h:1352:29: note: template<class T> bool flatbuffers::Verifier::VerifyBuffer(const char*)
template<typename T> bool VerifyBuffer(const char *identifier) {
^
include/flatbuffers/flatbuffers.h:1352:29: note: template argument deduction/substitution failed:
include/flatbuffers/flatbuffers.h:1421:37: note: candidate expects 1 argument, 0 provided
return verifier.VerifyBuffer<T>();
^
This commit fixes the BufferRef call site by adding a nullptr argument.
Unions own the NativeTable* value member because they need to destroy them
when the Union goes out of scope. Currently, the data is destroyed by calling
delete, which means that the member needs to be allocated with new. However,
making the allocation the responsibility of the client and the destruction
the responsibility of the Union can lead to potential errors. Adding a
Set function will ensure that the memory is allocated correctly so that it
can be deleted later.
From cl/142161569.
Change-Id: I4605f26d2749164819bfae0140e5fae08442b50a
* Changes for verifying a buffer dynamically using reflection.
* Fixing build issues on linux and applied code reformatting.
* Fixing the file order in cmake file that was messing up the macro based code inclusion.
Added tests for reflection based verification.
* Changes for verifying a buffer dynamically using reflection.
Fixing build issues on linux and applied code reformatting.
Fixing the file order in cmake file that was messing up the macro based code inclusion.
Added tests for reflection based verification.
* Incorporated the code review changes that were requested:
1. Changed the Verify function signature.
2. Changed the variable names to use snake_case.
3. Added better comments.
4. Refactored duplicate code.
5. Changed the verifier class so that it has the same size when compiled with or without FLATBUFFERS_TRACK_VERIFIER_BUFFER_SIZE macro.
* Setting FLATBUFFERS_TRACK_VERIFIER_BUFFER_SIZE and FLATBUFFERS_DEBUG_VERIFICATION_FAILURE through cmake for flattests so that it gets propagted to all translation units of flattests.
* Making the Verifier struct fields the same in all cases. Also reverting the target_compile_definitions change in cmake file because build machine on travis does not have cmake version 3.0 or higher which was the version when target_compile_definitions was added in cmake.
* Defining macros through cmake in a portable way using functions that are available in cmake 2.8.
This was causing build failures with tools dependent on Flatbuffers in Go.
E.g.
go/src/github.com/google/flatbuffers/go/sizes.go:50: undefined: unsafe in unsafe.Pointer
* support for grpc golang
* refactored grpc go generator
* added grpc-go test and refactored
* refactored idl_gen_grpc.cpp
* fixed grpc generate method name
* refactored flatc and fixed line length issue
* added codec to go lib and fixed formatting issues
* fixed spacing issues
The following changes have been made to the C++ codegen to enable writing generic code
that uses the Table and NativeTable types.
- Adds TableType and NativeTableType typedefs to NativeTable and Table structs.
- Adds GetFullyQualifiedName() to NativeTables if --gen-name-strings is set.
- Adds a static Pack function to Tables that simply calls the global CreateX functions.
See cr/140391505 as an example of improved usage.
From cl/140529288.
Change-Id: Idec137c16129e15c1783f94fabdcea24aeeaaef6
Introduce a "native_inline" attribute that can be applied on fields that are structs.
This results in NativeTable code generation that sets the struct "inline" rather than
storing it in a pointer.
From cl/140527470.
Change-Id: I208724f552b8b921b20923e0bf82f42cb3582416
I've faced an issue where I want to serialize UTF-8 emojis and FlatBufferBuilder::is_utf8 would return false on them.
I was not able to add the corresponding hexadecimal code because I don't understand how the whole thing works but what i've done is using the `mb_detect_encoding` function which handles correctly all supposed UTF-8 characters.
Also fixed structs not being created inline in the tutorial,
which would actually have bad consequences if used.
Change-Id: Idce215c61a1b24a297cee76f625052bb2722e970
* Make parameter order in jsdocs match function
When the order doesn't match the Google Closure Javascript compiler generates
warnings
* Prefix optional parameters with opt_
The Closure Compiler emits warnings when this isn't the case
* Java: emit "signed" equivalent of unsigned default value in generated code and add "L" suffix to long default value.
* Updated generated code
* Only convert ulong to "signed" equivalent. ubyte and ushort don't need specific handling as "user facing" type is int. uint need 'L' suffix as "user facing" type is long.
* Added missing cast to primitive type of default value which is in "user facing" type in builder.add<type>() calls.
* Do not cast default value to actual type in C#.
These are useful for streaming FlatBuffers. The functionality
ensures proper alignment of the whole buffer.
Tested: on OS X.
Bug: 27123865
Change-Id: Ic7d75a618c1bb470ea44c4dcf202ff71f2b3f4f1
Signed-off-by: Wouter van Oortmerssen <wvo@google.com>
This allows hashed string fields to be used for lookup of any
C++ objects, a pointer to which are then stored in the object
besides the original hash for easy access.
Change-Id: I2247a13c349b905f1c54660becde2c818ad23e97
Tested: on Linux.
Bug: 30204449
* Fix flatc generating incorrect java/c# code for namespace_test2.fbs. In code for TableInC, method referToA2() returned type SecondTableInA instead of NamespaceA.SecondTableInA.
* Updated generated code.
* Fixed indendation.
* Generate type traits for unions to map a type to the corresponding union enum value.
* Fixed break with union enum type traits when type is in a namespace.
* Fixed spacing and variable names in type traits generation to match style guidelines.
* Fixed spacing in type traits generation to match style guidelines.
* Regenerated test schema header.
Cmake issued a warning when the variable is in quotation marks
in an if statement.
Visual Studio upgrades constants to int and issues a truncation
warning, so inserted a cast.
Change-Id: I60cdcb5c2565cd5e97f80b9c2ff1e6abc32b1deb
Tested: Builds without warning on VS2015.
It is a common practice to put all the compiled classes into a
dedicated folder in order to:
- avoid to mix the code and the complied classes
- to allow a simple way to remove the complied classes
Add log of java version in order to make it explicit to the end user.
It is a common practice to put all the compiled classes into a
dedicated folder in order to:
- void to mixing the code and the complied classes
- to allow a simple way to remove the complied classes
Similar to what protobufs does with its `Message` interface, introduce here such interface and create a generic `GetRootAs` method to deserialize a flatbuffer.
They were overloaded, but also had default arguments, so would
become ambiguous when used with few arguments.
Change-Id: Ifac7f3ea3a6391d971dfeda8e33129c8c38d6f12
Tested: on Linux.
This was a frequent source of confusion, since in all implementations
the data doesn't start at offset 0 in the buffer.
Change-Id: I045966e65928e9acd9def84e215914ecb5510653
The new line in the set(CMAKE_CXX_FLAGS...) line was getting insert
verbatim into the Makefile. Makefiles don't like new lines in their
assignment operators.
The newline escape fix works for CMake 3.0 and above, but since
we support 2.x, we need to use the legacy solution, which is to
split into two separate statements.
Tested: cmake -G"Unix Makefiles" works now on Mac.
Change-Id: I6f4655981b85087c4760c3d26ed0c97c4469ba93
It would write 64bits offsets instead of 32bit ones, and update
the vtable pointer before the fields were processed.
Change-Id: I0c0fa942bbd3b42839294f5653ba8fa048612624
Tested: on Linux.
Minimal, in the sense that this will only allow flatbuffers.h +
generated code to work. Everything else (tests, parsing, reflection
etc.) may still not compile with stlport.
Functionality has been reduced, some utility functions are not
available.
Tested: on Linux (no stlport), Android (stlport).
Change-Id: I3f8b6a88258c07d78964dd455fb9f99f65266301
This avoid to put the pom.xml file into the source directory. Normally the pom file is in a parent (/parent) folder and it is not mixed with the java source code.
An other thing is: this will make import of the project more easy from a IDE.
The side effect is that the target folder where maven build artifacts will move from the <flatbuffers>/java/target to <flatbuffers>/target therefore the gitignore file has been updated in consequences.
This is useful because many JSON generators will sort the fields,
cause X_type to follow X.
Change-Id: I00ef3ac05418224fc05aee93e6b3b3597e73ffe3
Tested: on Linux.
Bug: 29221752
Also fixed most codegenerators using the wrong namespace when multiple
namespace were used in a file, with some files not being generated.
Change-Id: Ib42969221239d7244e431cbd667ef69200fc415f
Tested: on Linux.
Bug: 29338474
Java developer are mostly comfortable with maven project structure. One one the main concept behind maven is convention. If you follow the maven project convention then your development team will get more effective as they now this project structure and can easily find the production code versus the test code.
In this pull request I have structured the java project around 2 main parts:
* the `flatbuffers` project. This project is the api / lib project and contains the test code structure + an example of code generation for testing. This avoid to commit generated code. Pre-configure JUnit for test driven development and make this project OSGi compliant.
* the `jmh` project. This project aims to provide a placeholder for micro-benchmarking. JMH is a 'de facto' standard for micro benchmarking you can find more details here: http://openjdk.java.net/projects/code-tools/jmh/
For now I didn't move the JavaTest class but it could be a next step with a migration to the JUnit framework.
The only impacts are the move of the class and the project structure => no code change.
add more items to the corpus (taken from the ClusterFuzz-generated corpus).
(mirrored from cl 122690857)
Change-Id: I98c7672169a9a27fe18bedae8501e6fb85a0681d
GetObject and GetMutableObject are similar to GetRoot and GetMutableRoot,
and can be useful when wanting to access data that has just been created.
Unfortunately there is a danger in using these methods, as it is possible
that the buffer reallocates which will invalidate the pointers.
This avoids the following allocations over repeated use:
- A ``CharsetEncoder``
- A byte array to contain the contents of the string
- A wrapper, via ``ByteBuffer#wrap``, for the previously mentioned byte
array
This also removes a copy of the temporary byte array. This extra copy
is needed because ``String#getBytes`` returns a byte array and its
length must match the exact size of the contents.
This implementation requires that the builder retain a ``ByteBuffer``
and ``CharEncoder``. This is considered slower only for users that
will only allocate a single string in their buffers or none at all.
The improvement is especially potent if the builder is constantly reused
by its caller. This also eliminates the penalty for users that do not
use strings at all since the cost of allocating these fields is
now amortized.
The only allocation left in this code is a ``CharBuffer`` wrapper. It's
possible to eliminate this allocation using an additional copy but need
further profiling to see which is the bigger problem.
Removes the following allocations:
- ``CharsetDecoder`` is reused between calls
- ``CharBuffer#wrap`` removed in favor of heap
based char buffer that is reused
- Temporary ``char[]``, an intermediate copy inside ``StringCoding``
- Another ``char[]``, this is needed because ``StringCoding`` uses
a ``CharBuffer`` internally but returns a ``char[]``. Extra
characters need to be trimmed so this means yet another allocation
- Yet another ``char[]`` directly from ``__string`` for non-heap
based buffers
Removes the following copies
- No copy is performed to trim the allocation since a ``CharBuffer``
is used directly
- For non-heap based byte buffers, removes the copy that was
previously done in the __string function
This does need to get the TLS entry which implies at least some
contention on the thread object table and a fence.
A previous commit that added "f" for C++/Java/C# would break
on gcc of constants like 3f, which are now output as 3.0f
Tested: on Linux
Change-Id: If9cabbe3c6d6948a5050b8b123bda9c06e181f52
CreateUninitializedVector was performing the following actions:
1. call StartVector.
2. call make_space, and set buf to point to the reserved space.
3. call EndVector.
The problem is that a call to EndVector can ultimately call make_space, which
if the buffer is full, will cause a reallocation, invalidating the value stored
in buf. So setting buf needs to be delayed until after EndVector.
The following code, when run under valgrind shows a write to free'd memory before
the change, but no such error after:
int main()
{
flatbuffers::FlatBufferBuilder fbb(128);
char *buf = nullptr;
fbb.CreateUninitializedVector(128, &buf);
*buf = 0;
}
This overflow could allow malformed FlatBuffers to pass the verifier.
Change-Id: Ia993299a761b00b93e53e8aff2689b631755763c
Tested: on Linux.
Bug: 27903580
To support the use case described in issue google/flatbuffers#3826, a new command line option --gen-name-strings
has been added, which will cause a static GetFullyQualifiedName function to be added
to the C++ output for tables/structs.
Currently in order to get a value type of [ubyte] in PHP, iteration is
necessary which is not efficient. Helper __vector_as_bytes has been
implemented in order to return the byte arrays in PHP efficiently.
Appropriate methods also been added to use aforementioned method to return
the byte array. (e.g. get*Bytes()).
The reason the methods are named get*Bytes() and not for instance
get*ByteArray() is the fact that PHP doesn't support byte arrays and the
binary safe string implementation in PHP is used to simulate byte arrays
and since there is chance for PHP users to confuse this with PHP arrays
the name get*Bytes() has been chosen.
In the future __vector_as_bytebuffer() method can also be implemented to
return PHP implementation of ByteBuffer.
Certain architectures, such as ARM, use unsigned chars by default
so require the `-fsigned-char` for certain value comparisons to
make sense and in order to compile.
This is the first step in RPC support. Actual code generation
to follow.
Change-Id: I96c40fec3db671d100dd9eb509a71c5cbe55bfb2
Tested: on Linux.
Bug: 20122696
There was no way to pass an already-encoded string to
`builder.CreateString` in Python 2.7:
- Passing a `bytearray` raised a TypeError because `bytearray` was not
recognized as an instance of `compat.binary_type`.
- Passing a utf-8 encoded `str` would cause the string to be
double-encoded, because `compat.string_types = (basestring,)` and
`basestring` is the base class of `str` and `unicode`, so the logic
would never reach the `elif isinstance(s, compat.binary_type)` case.
- Converting a utf-8 encoded bytearray to `bytes` like
`builder.CreateString(bytes(encoded_string))` does not work because
in Python 2.7, bytes is just an alias for `str` so it behaves as
above.
This change allows either `bytes` or `bytearray` as an already-encoded
string to be passed to `CreateString` in versions of Python that support
`bytearray`, and falls back to `str` in older versions.
In Python 2, it restricts unencoded string types to `unicode`, so `str`
can be used as an encoded, binary representaiton.
The code generator was assuming all declarations for the current
file sit in the same namepace. Now uses the "on demand" namespace
switching we had for the forward declarations.
Also fixed a bug related to namespace lookup.
Change-Id: Ib54a3efbc752cbb9590302fa0707c0c73448db3d
Tested: on Linux.
Test was testing for a field to be None, even though it is populated
in one particular instance.
Change-Id: Id19848d01498d4c0a684efdb2bf49ff12c5b6758
Tested: on Linux.
When built for release, builds were failing with unused
variable warnings, since they were only used in `assert()`
calls.
I added explicit void casting to any potentially unused variables.
Change-Id: I9947ba46891fdda5aa925caa950642dedd4e009f
Adding an API reference for the supported languages.
General docs cleanup, including a new `tutorial` section that
supports all of the supported languages.
Added samples for each supported language to mirror the new
tutorial page.
Cleaned up all the links by making them `@ref` style links,
instead of referencing the names of the generated `.html` files.
Removed all generated files that were unnecessarily committed.
Also fixed the C# tests (two were failing due to a missing file).
Bug: b/25801305
Tested: Tested all samples on Ubuntu, Mac, and Android. Docs were
generated using doxygen and viewed on Chrome.
Change-Id: I2acaba6e332a15ae2deff5f26a4a25da7bd2c954
Binary file arguments to flatc have to be preceded by -- to
identify them, forgetting this however results in them being
attempted to be parsed as schema/json, with cryptic errors.
This instead gives an error if 0 bytes are contained in your
text file.
Bug: 22069056
Change-Id: I226bf651dcb016f18d7c8ffadcf23466a1fc0b87
Tested: on Linux.
The parser state generated from the .proto conversion process is
not exactly the same as what you get by parsing the generated
schema, which can cause problems. This check enforces that you
first convert the .proto, then generate code from the new schema.
Change-Id: I04b53af9288d87e256d1cc109388332fefb3a09f
Tested: on Linux.
src\reflection.cpp(297): warning C4267: 'argument' : conversion from 'size_t' to 'flatbuffers::uoffset_t', possible loss of data
sizeof() was promoting the type from uoffset_t to size_t.
Cleaned up a few warnings to allow VS2012 to compile idl_parser and idl_gen_text (for exporting binary protobuf blobs as JSON) cleanly under static analysis.
Added a check for a preprocessor definition that can be set if the platform you're building for doesn't support any notion of absolute path resolution/realpath()/etc.
When passing a unicode string to builder.CreateString, the
default encoding assumed all characters can be encoded
using ascii. Added a fix so a user can specify the
encoding and how to handle errors when creating strings.
This is implemented as a template function, since Table::CheckField
is not reachable by subclasses of Table (private base class).
Change-Id: I1ed4d47ce7cb672460ccab61cf7442eb9136b0f1
Tested: on Linux.
Bug: 26273432
This to allow the code to run on a greater range of build
configurations (that don't allow exceptions/RTTI).
If anyone ever doubts the usefulness of exception handling,
please show them this commit.
Change-Id: If7190babdde93c3f9cd97b8e1ab447bf0c81696d
Tested: on Linux.
Removes the bounds checking on the ByteBuffer Get/Put operations. Can be dangerous when used with UNSAFE_BYTEBUFFER but results in increased performance. Use at your own risk!
Fix for Issue 1741
Minor bug where python versions 2.7.x where x < 5 do not support
unpacking from memoryview objects. Versions 2.7.5 and above will
use memoryview while 2.7 versions below 2.7.5 will use buffer
objects.
Manual testing was performed on versions 2.7.5 and 2.7.2 to
confirm both worked correctly.
It's slightly faster to convert the value to signed value in PHP as
opposed to use pack and unpack.
For 1M get operation the difference is:
getShort in 3.3272678852081 seconds
getInt in 3.8338589668274 seconds
getLong in 5.6381590366364 seconds
getLong (neg) in 5.6149101257324 seconds
vs
getShort in 2.7564418315887 seconds
getInt in 3.1612701416016 seconds
getLong in 3.1369340419769 seconds
getLong (neg) in 3.1478710174561 seconds
And since pack("P") and unpack("q") has been removed now ByteBuffer
works for PHP >= 5.4
I updated idl_gen_general.cpp to add support for generating a Get Bytes
method for a vector to the generated C# source code. Given a byte vector
field named Foo, a method named GetFooBytes() will be generated in the
C# source code that will return an ArraySegment<byte> value referencing
the vector data in the underlying ByteBuffer.
I added a method to Table.cs named __vector_as_arraysegment that is used
by the code generated by the change to the C# generator.
__vector_as_arraysegment will take the offset of the vector and will
return the ArraySegment<byte> value corresponding to the bytes that
store the vector data.
I updated FlatBuffersExampleTests.cs to add tests to validate my
implementation of Table.__vector_as_arraysegment. I added tests to
demonstrate that the bytes for the monster's name can be extracted from
the underlying byte array. I also added tests to show that
Table.__vector_as_arraysegment returns a null value if the vector is not
present in the FlatBuffer.
I used the updated flatc.exe program to regenerate the C# source files
for the MyGame example. The new Monster class includes the GetXXXBytes
methods to return the byte arrays containing data for vectors.
Simply running clang with -stdlib=libc++ requires a manual link against
libc++abi on Linux. This is documented in the libc++ documentation:
http://libcxx.llvm.org/
Tested on Arch Linux, using clang 3.7.0 RELEASE.
Signed-off-by: Perry Hung <iperry@gmail.com>
It used to be such that later schemas could depend on earlier
schemas. This was a convenience from days before include files
were implemented. Nowadays they cause subtle bugs rather than being
useful, so this functionality has been removed.
You now need to explicitly include files you depend upon.
Change-Id: Id8292c3c621fc38fbd796da2d2cbdd63efc230d1
Tested: on Linux.
Besides making the generated code looking a lot more readable,
it also allows you to use these offsets in calls to
Table::CheckField, to see if a field is present in a table.
Change-Id: I1b4cc350c4f27c4e474c31add40c701ef4ae63b2
Tested: On Linux.
Also checks for nesting of objects in vector construction.
This avoids common errors in FlatBuffer construction.
Change-Id: I5507c5d767684e20e94883a92448f05acefba4d6
Tested: on Linux.
Simply running clang with -stdlib=libc++ requires a manual link against
libc++abi on Linux. This is documented in the libc++ documentation:
http://libcxx.llvm.org/
Tested on Arch Linux, using clang 3.7.0 RELEASE.
Signed-off-by: Perry Hung <iperry@gmail.com>
It used to be such that later schemas could depend on earlier
schemas. This was a convenience from days before include files
were implemented. Nowadays they cause subtle bugs rather than being
useful, so this functionality has been removed.
You now need to explicitly include files you depend upon.
Change-Id: Id8292c3c621fc38fbd796da2d2cbdd63efc230d1
Tested: on Linux.
Besides making the generated code looking a lot more readable,
it also allows you to use these offsets in calls to
Table::CheckField, to see if a field is present in a table.
Change-Id: I1b4cc350c4f27c4e474c31add40c701ef4ae63b2
Tested: On Linux.
Also checks for nesting of objects in vector construction.
This avoids common errors in FlatBuffer construction.
Change-Id: I5507c5d767684e20e94883a92448f05acefba4d6
Tested: on Linux.
Fixes a bug where the logic to determine when to use a C# enum flags
both enums and vectors of enums. This causes the C# generator to
generate code that doesn't compile for tables that contain vectors of
enums.
The fix also consolidates type generation functions a bit and adds
some additional casting functions for clarity.
Also, added missing build scripts for flatc on Windows and fixed
numerous issues in the flatbuffers/android/jni/include.mk makefile.
Tested:
Verified dependent project (motive) can generate flatbuffers headers
with these changes on Android.
Bug: 25780616
Change-Id: I73f0dec8f7a6a7eb4e71951a8f1cd3e0d343a9ff
* codegen for all basic features: WIP (probably implemented all basic feature)
* JSON parsing: NO
* Simple mutation: NO
* Reflection: NO
* Buffer verifier: NO (will be add later)
* Testing: basic: Yes
* Testing: fuzz: Yes
* Performance: Not bad
* Platform: Supported Linux, OS X, Windows (has 32bit integer limitation)
* Engine Unity: No
flatc --php monster_test.fbs
<?php
//include neccessary files.
$fbb = new Google\FlatBuffers\FlatBufferBuilder(1);
$str = $fbb->createString("monster");
\MyGame\Example\Monster::startMonster($fbb);
\MyGame\Example\Monster::addHp($fbb, 80);
\MyGame\Example\Monster::addName($fbb, $str);
$mon = \MyGame\Example\Monster::endMonster($fbb);
$fbb->finish($mon);
echo $fbb->sizedByteArray();
PHP 5.4 higher
Currently, we do not register this library to packagist as still experimental and versioning problem.
If you intended to use flatbuffers with composer. add repostiories section to composer.json like below.
"repositories": [{
"type": "vcs",
"url": "https://github.com/google/flatbuffers"
}],
and just put google/flatbuffers.
"require": {
"google/flatbuffers": "*"
}
* PHP's integer is platform dependant. we strongly recommend use 64bit machine
and don't use uint, ulong types as prevent overflow issue.
ref: http://php.net/manual/en/language.types.integer.php
* php don't support float type. floating point numbers are always parsed as double precision internally.
ref: http://php.net/manual/en/language.types.float.php
* ByteBuffer is little bit slow implemnentation due to many chr/ord function calls. Especially encoding objects.
This is expected performance as PHP5 has parsing arguments overhead. probably we'll add C-extension.
Basically, PHP implementation respects Java and C# implementation.
Note: ByteBuffer and FlatBuffersBuilder class are not intended to use other purposes.
we may change internal API foreseeable future.
PSR-2, PSR-4 standards.
Implemented simple assertion class (respect JavaScript testcase implementation) as we prefer small code base.
this also keeps CI iteration speed.
we'll choose phpunit or something when the test cases grown.
This allows you to query the generated target for those properties
later, so dependent modules can refer to the directories to, for example,
add them to the include_directories.
Change-Id: I7a6bd34c5c1d08e2ea69b5ad845223297cad1159
This function is designed to be called by projects that use FlatBuffers.
It takes a list of FlatBuffers schemas and a list of schema include
directories, and generates build rules to create generated headers
and/or binary schemas for those files, as well as optionally copying
all of the text schemas into a directory for your use (for example if
you need to parse text schemas within your app).
See function comments for more details.
Change-Id: I181684b916e91d965e14849d8f83ec2c83e9a4a4
local-source-file-path does not expand to correct file paths in
some circumstances so some users override it. Therefore
flatbuffers_header_build_rules has been modified to generate rules
that expand LOCAL_SRC_FILES values with flatbuffers_header_build_rules.
Also, this overrides local-source-file-path to allow nest projects
to build when NDK_OUT is set.
Tested:
Verified a dependent project continues to build.
Bug: 25673744
Change-Id: Ic90186fe96d6e4533f9f3b7ca9ef78084de08a7e
When $(MAKE) is expanded via a macro "make" doesn't know whether
$(MAKE) refers to an instance of the make app and therefore doesn't
pass job server arguments down. This change adds the + prefix to the
command in the receipe in order to indicate $(MAKE) is another instance of
make.
Tested:
Verified flatc builds in parallel from Android builds.
Change-Id: I9f2f4f9680b818fdda7420a75a8bfa995b4644db
+ Add state to the Builder object to track if we are inside a table,
and if we are finished building the buffer.
+ Use this data to check that a buffer is being built correctly.
+ Raise an exception if a buffer is not being built correctly.
+ Test that the exceptions happen as expected.
Based on d236dea.
+ Add state to the Builder object to track if we are inside a table,
and if we are finished building the buffer.
+ Use this data to check that a buffer is being built correctly.
+ Panic if a buffer is not being built correctly.
+ Test that the panics happen as expected.
Based on d236dea13d.
* Added the ability to create a build target for generated headers.
* Made it possible for generated header targets to depend upon each
other or arbitrary build targets.
Tested:
Verified some pretty complex libraries with numerous flatbuffer schema
dependencies build using this macro on Linux with the NDK.
Bug: 25188384
Change-Id: I846855a50808e58c34cdf7086e93e7ea0df69e0d
Because they are represented as `const T *` in the Vector template,
the sizeof(const T *) was accidentally used instead of sizeof(T).
Change-Id: Ib4dc73e1d21396ba2e30c84e5e229c4147204bb1
Tested: on Linux.
This adds an optional argument to generated getters for string fields to
specify the encoding type and overloads the createString() function. It's
now possible to use either JavaScript UTF-16 string objects or C-style
UTF-8 byte arrays (Uint8Array) for string data.
JavaScript uses UTF-16 but FlatBuffers uses UTF-8. This commit tests the code
that does the conversion between the two encodings. The last entry in the array
is tricky because each code point actually requires two UTF-16 code units,
unlike the other examples. The current JSON output of flatc actually handles
this case incorrectly (it generates invalid JSON with UTF-8 code units). The
generated JavaScript code passes these tests fine, however.
This adds a JavaScript language target. The generated JavaScript uses Google
Closure Compiler type annotations and can be compiled using the advanced
compilation mode, which performs type checking and optimizations such as
inlining and dead code elimination. The generated JavaScript also exports all
generated symbols for use with Node.js and RequireJS. This export behavior
can be turned off with the --no-js-exports flag for use with Google Closure
Compiler.
The pop_back function was added to strings in C++11 and it appears not
all compilers we target support it. The call to pop_back has been
replaced with a call to erase.
Tested on Linux. All unit tests pass.
Ensure the size of the buffer being checked is bigger than the element
of the buffer being checked. The buffer can be triggered when, for
example, the buffer is of length zero and we are checking for:
Verify<uoffset_t>(buf_)
The condition above should fail.
Also, remove execute permissions on several source files.
Tested: Builds on Visual Studio 2012, and Linux.
Change-Id: Idaacb2ae8eba98ce2974218c2ab840e97a1d67e9
Usually generators add @SuppressWarnings("all") to generated Java
classes to prevent IDEs from complaining about unused imports, etc.
Solving used imports seems pretty hard with current generator logic so
IMO this is the next best thing.
Yes, it’s appended to import block but that is the block that gives
these warnings in the first place.
include.mk causes absolute paths to be generated, which doesn't
work on Windows (make doesn't like : in paths).
Change-Id: Ib06c9581620bde1e8452e0dd887d3986d517c4f5
Tested: on Windows, Linux.
When creating a CreateXxx(...) method for a simple table type,
enum-type fields with a non-zero default must have an explicit
cast for the respective argument default value, because in C#,
there is an implicit cast from int to an enum only for 0.
Also, added an example of such type into the example monster_test
type, so that we test this feature.
Also, removed the flag from the Android makefile.
Tested:
Verified flatc builds successfully on Linux and regenerated the docs.
Change-Id: I7140daa10b7cb9a29b5ffd63c6b20489e72a5899
This allows you to use these functions with a flatbuffer whose root
table type does't correspond with the root table type of the schema.
If you don't specify the table name, it will use the root table from
the schema by default (mimicing the current behavior).
This pull request should fix#232.
Pass field name instead of struct name as nameprefix for nested structures. Continuously concatenate nameprefix arguments to support deeper nested structures.
Before this change, requesting a large initial allocation could cause the
backing store to grow to an unaligned size. Since memory inside vector_downward
is relative to the end of the buffer, this then caused all memory in the buffer
to be misaligned and also misaligns any further loads and stores. Misaligned
loads and stores are undefined behavior and don't work in environments such as
emscripten (a JavaScript to C++ compiler).
On Android builds, set FLATBUFFERS_FLATC_ARGS to change the arguments
passed to the flatc cmdline. Do this in your Android makefile where
you include the flatbuffers include.mk (before or after).
For example FLATBUFFERS_FLATC_ARGS=--gen-mutable
The default value is --gen-includes although since that is deprecated
we could just make the default value blank.
Change-Id: I79fb35f50c3e21bbef18ad40ad3559cb026ffe8e
Also add a new `insideObject` boolean to the Builder to track whether an
object is currently being constructed. This fixes a bug with objects
that have zero fields.
--gen-includes is now deprecate but still accepted.
--no-includes can be used instead when needed.
Change-Id: I2cd46d193032b9b7c31c76c6f655e9333d3a393a
Tested: on Linux.
Normal behavior is to not output fields that happen to have
the default value, since those will be reproduced anyway
when turned into a FlatBuffer binary. This however can be problematic
when using JSON to interop with other system since they might not
know this default value. This flatc option (and also flag
to GenerateText) will force those fields to be output anyway.
Tested: on Linux.
Implement code generation and self-contained runtime library for Python.
The test suite verifies:
- Correctness of generated Python code by comparing output to that of
the other language ports.
- The exact bytes in the Builder buffer during many scenarios.
- Vtable deduplication correctness.
- Edge cases for table construction, via a fuzzer derived from the Go
implementation.
- All code is simultaneously valid in Python 2.6, 2.7, and 3.4.
The test suite includes benchmarks for:
- Building 'gold' data.
- Parsing 'gold' data.
- Deduplicating vtables.
All tests pass on this author's system for the following Python
implementations:
- CPython 2.6.7
- CPython 2.7.8
- CPython 3.4.2
- PyPy 2.5.0 (CPython 2.7.8 compatible)
When accessing a union field, we should return the same object type
as was given to the method, i.e. the parameter should have a generic
type for any Table-derived type. This way, we do not need to make
superfluous casts (which also reduce type safety) like
var myUnionType = (MyUnionType)buff.GetUnionField(new MyUnionType());
when we can do just
var myUnionType = buff.GetUnionField(new MyUnionType());
Change-Id: Idac1b638e46cc50b1f2dc19f10741481202b1515
Allows adding an already encoded UTF-8 string directly without
having to convert to a ``String`` first.
Change-Id: I23f9c738eec18fd35f4c14f58dbd0f6cf0970dc7
Also made proper namespacing work for enums.
You can now say namespace.MyTable as the type when declaring
a field that refers to a type in a different namespace.
Previously, it would work just referring to MyTable, however
with the recent commit fixing namespaced types this now
is ambiguous.
Change-Id: Ieaa3f4ac1662b8c4dc1f16e1898ea3cdb02e10fd
Tested: on Linux.
In C#, plain field accessors should not be nonparametric methods
but should be standard property getters.
The accessor methods with parameters were renamed to `GetXxx`
because a method cannot be named identically to a property.
Also, `ByteBuffer.Position`, `FlatBufferBuilder.Offset` and
`FlatBufferBuilder.DataBuffer` are now properties instead
of nonparametric accessor methods, for more idiomatic C# style.
This is a breaking change, all client C# code accessing these
fields needs to be changed (i.e. remove those `()` or add the
`Get` prefix).
Issue: #77
Change-Id: Iaabe9ada076e5ea2c69911cf6170fdda2df3487e
By default, disallow reading binaries that do not correspond to
the schema file_identifier. Override with --raw-binary
Change-Id: I24822c0e4c80dab4e5a238ae2b6e601a09025c27
Tested: on Linux.
This commit contains the first step in providing mutable FlatBuffers,
non-const accessors and mutation functions for existing fields generated
from --gen-mutable.
Change-Id: Iebee3975f05c1001f8e22824725edeaa6d85fbee
Tested: on Linux.
Bug: 15777024
If an IDL file uses DOS-style EOLs (CR+LF), line comments need to
ignore the second linebreak character, otherwise, as is currently
the case, the parsed documentation comment includes a trailing `\r`
character, which is then output verbatim into the output source
code by flatc.
Change-Id: I39591631995a980622d20a4a32315178b33f18f6
added bii-travis.sh and readme.md files; added build status section with travis and biicode build
systems
Change-Id: I93acd1ca9497416ade6293d63b5311c4c31b880f
Enums should not be (badly) emulated with classes in C# but should
use native C# enums instead. Java implementation made an explicit
choice not to use the (more complex) Java enums, but C# enums are
just light-weight syntactic coating over integral types.
Fixes issue #171.
Change-Id: I9f4d6ba5324400a1e52982e49b58603cb7d7cca7
Builder has a new CreateByteString function that writes a
null-terimnated byte slice to the buffer. This results in zero
allocations for writing strings.
- Add parse handling of .proto struct and field doc comments (enums and
their values were already handled)
- Add FBS generation handling of doc comments for structs, their
fields, enums, and their values (requires linking idl_gen_general in
the test binary build)
- Tested using test.proto|golden with doc comments added. Xcode run of
flattest passes.
Change-Id: Idff64dd8064afba227174ab77d2c7be22d006628
Tested by regenerating all tests/ generated sources; note that only
Monster.java changes. Ran flattests as well.
Change-Id: I65b6ea7d208b0ccd6a0b34761162fed6ba391fc5
We were looking for 'flatc' in motive/bin/Debug/flatc,
on Mac. It's actually built to flatbuffers/Debug/flatc.
Tested: OS X Yosemite, ndk-r10d. Also tested on Linux.
Change-Id: I9f1ecfe00c5f42fd9b6808b5a5da1c920487a4c2
(Java doesn't support unsigned types).
ubyte/ushort return as int
uint returns as long
(all with correct masking)
ulong still returns as long, as before.
Tested: on Linux & Windows.
Bug 17521464
Change-Id: Id6bc8f38fc8c1a2f4e6733c6980dc6b6e322b452
BitConverter was excessively slow since it allocates a byte array
at each access.
Bug: 18702381
Change-Id: I47be9c38e1d04287ba4c10bc369848f3e13a2a2f
Tested: on Windows.
There is a bug in creating a C# table when it includes a field of type 'BOOL'. The problem is that the generate C# code is as follows:
"bool SampleValue = 0;"
This will fail to compile, because in C# this fails, it needs to be generated as:
"bool SampleValue = false;"
The error is in line ~510
Change-Id: I77f6eea0f269b0540dbeb462602fc447cb69237d
This function gets around the inefficiency of populating a [ubyte] vector
byte by byte. Since ubyte vectors are probably the most commonly used type
of generic byte buffer, this seems like a worthwhile thing to create a
fast path for.
Benchmarks show a 6x improvement in throughput on x64.
There is a new test verifying the functionality of the function.
Change-Id: I82e0228ae0f815dd7ea89bf168b8c1925f3ce0d7
1. Random table fields are now marked deprecated. The deprecation rate is adjustable, default is ~10% of fields.
2. Vector type previously produced a dummy field, now a trivial ubyte vector is generated instead.
3. The fuzzed "instances" were previously generated with identical data, now they are randomized: I suppose such was the intent from the beginning, just wasn't implemented.
I ran the new test with a few different settings and random seeds (but only on Windows + VS2015), and it always passed.
Change-Id: I2014686b6680aec24049ccda3f6deffe5205a83e
* Tests for Release feature.
* Check vector_downward.buf_ before passing to deallocator.
* Assertions.
* Shared test between unique_ptr and GetBufferPointer()
* Unnecessary using directives.
* Reallocate vector if released on clear operation.
* Use allocator attribute.
* Renamed `Release()` to `ReleaseBufferPointer()`
* For consistency with `GetBufferPointer()`
* Updated documentation for ReleaseBuffer.
Change-Id: I108527778e56ae5127abf9e5b1be6b445ad75cb7
build_apk.sh assumed pre-releases would be sorted towards the end,
but instead they end up at the front (sort -n) causing the api
selection loop to halt early.
bug: 19213196
Change-Id: I210a18c16e81880229f154c4613f8b97d90ac9bd
tested: on Linux.
FlatBuffer schema files can now optionally specify a hash attribute that
will allow someone writing json files to enter a string to be hashed
rather than a specific value. The hashing algorithm to use is specified
by the schema.
Currently the only algorithms are fnv1 and fnv1a. There are 32 bit and
64 variatns for each. Additionally, a hashing command line tool was
added so that you can see what a string will hash to without needing to
inspect the flatbuffer binary blob.
Change-Id: I0cb359d0e2dc7d2dc1874b446dc19a17cc77109d
Added a call to $(strip) to the parameters of
`flatbuffers_header_build_rules` so that newlines can be used when
passing arguments to it.
Change-Id: Ie2149acebcef91d28ce2cb4bfd204a209b4c4e2f
Previously, if you were re-using instances of FlatBufferBuilder
and an earlier buffer would use a 64bit item where later ones
do not, you could be wasting space.
Change-Id: Ic8090a38f97ce73194e991ba72bcfae74a7ace9f
Tested: on Linux.
getRootAs..() function now has a second implementation that
accepts an existing object to allow object reuse, much like
all other methods that refer to objects.
Change-Id: Iffef567c903a130761ef7de98867e5465d29a04d
Previously Android.mk only had a rule for the Flatbuffers test. There
is now an empty module definition so that flatbuffers can be included
as an Android module in other projects.
Additionally, android/jni/include.mk has been added which contains
some utility functions that can be used by projects using Flatbuffers
to generate header build rules and set up dependencies.
A sample project has been added to the samples directory to
demonstrate how to use flatbuffers with Android.
Tested by compiling Android project on Linux.
Change-Id: I25d6da40f6531777b22f7371187e0a2f4e903ad4
The satellite data of the ``ByteBuffer`` cannot be modified in
any way and stay thread safe in the presence of concurrent readers.
This implementation is simple and does introduce an allocation, however
without it multiple readers will quickly and continuously encounter
``IndexOutOfBoundsException`` exceptions.
An alternative, but possibly more controversial, implementation would
be to use ``Unsafe``. Using ``Unsafe``, it's possible to do an
array copy with a provided buffer index.
Change-Id: I851d4034e753b3be2931ee2249ec2c82dde43135
optimization would cause vtable fields from previous tables to be written.
Bug: 19046968
Change-Id: I781f7bcbceeaec0b499d4f1e4e5e8a1e750e0707
Tested: on Linux.
If the schema didn't contain a namespace, paths would contain a
leading /, causing files not to be written.
Change-Id: I508772cbf6d18d464ef7d9f8842d0dbff14358a3
Tested: on Linux.
Bug: 19067493
Unless in --strict-json mode.
Also added strict_json option to the parser, which in
addition controls if field names without quotes are allowed.
Change-Id: Id56fe5c780bdb9170958050ffa8fa23cf2babe95
Tested: on Linux.
https://github.com/google/protobuf/blob/master/java/pom.xml
This isn't good enough to publish to Maven Central but will at
least allow users to publish to their local maven repository
using 'mvn install'
Change-Id: I91ea146cf7c5263fcf5d9823f70bb1ef0158f9a6
Tested: 'mvn install' runs succesfully and produces a .jar
If your C# runtime environment supports unsafe mode, you can use
the #define UNSAFE_BYTEBUFFER setting and build the FlatBuffers assembly
in unsafe mode for greatly increased performance.
Tested: Tested FlatBuffersTest on Windows using VS2010 with both safe
and unsafe versions. Added ByteBufferTest to test the byte reversing
functions.
Change-Id: I21334468b339334f9abf4317e6291b648b97f57b
People sometimes accidentally inherit from these types.
Bug: 18224703
Change-Id: Ia09489a834ac4941f9b4a46f240cbdcf456f03a1
Tested: on Windows and Linux.
This is such that if you mis-spell an attribute, it doesn't get
silently ignored.
Bug: 18294628
Change-Id: I10013f5b2a21048b7daba2e9410678f528e09761
Tested: on Linux.
Added extra check to ensure memcmp gets called with a size that is
guaranteed within range of the buffer. This wasn't a real problem,
but stops address sanitizer from complaining.
See:
https://github.com/google/flatbuffers/issues/88517506b4e1 (commitcomment-8265231)
Change-Id: I7de24da2d36d973e154f92eeb3e093070886037f
Tested: on Linux
Noticed a memory read that isn't big-endian safe. Was somewhat
benign in that it would have simply caused vtable duplication
when constructing a FlatBuffer on a big-endian machine.
Change-Id: I5de3a2bb3ce6912fdd845ed40668719794920cac
Also fixed Go unit tests not being up to date with recent schema
changes.
Change-Id: I42e619f9c5ea05f6f937c68a5c8a92462c46bce3
Tested: on Linux and Windows.
Accessors and constructors now take enum types rather than ints.
Bug: 16570507
Change-Id: I4b50fd64ad2e662ea2481bc0ccea784326fb31c0
Tested: on Linux and Windows.
Looks like MSVC on Windows and like gcc everywhere else.
For enhanced IDE clickability.
Bug: 17208371
Change-Id: Ie3e02658fccd3edfd464b4bacf4bc68c613a8570
Tested: on Linux and Windows.
Also made the C# implementation support unsigned types, and
made it more like the Java version.
Bug: 17359988
Change-Id: If5305c08cd5c97f35426639516ce05e53bbec36c
Tested: on Linux and Windows.
This was only causing compiler errors on certain compiler configurations.
Change-Id: I110fb8c896f74aae7ef739e9a29c636393dbbde2
Tested: on Linux and Windows.
Also cleaned up ByteBuffer usage in general: ByteBuffer.position
now universally indicates the start of a ByteBuffer.
Change-Id: Ic4bfb98f9df9501b8fc82de2c45db7d7311135ac
Tested: on Linux.
Include C# codegen in flatc and .NET FlatBuffer access via the
FlatBufferBuilder class
Tested: on Windows.
Change-Id: If5228a8df60a10e0751b245c6c64530264ea2d8a
Added FLATBUFFERS_BUILD_TESTS option which can be used to disable
the build of flatbuffers tests and samples.
Tested:
Verified tests and samples are no longer built on Linux when
FLATBUFFERS_BUILD_TESTS=OFF.
Change-Id: Ic23ab827849ba2c4481de9ca86adc1ab8e6b828c
Readonly ByteBuffers and Direct ByteBuffers don't have a backing
array, and thus can't be used for writing FlatBuffers (though
they are fine for reading).
Change-Id: I4d7b9cc222b96161d0f8e92f34588bd3e0e38034
Tested: on Linux.
Since part of it is based on the filename, which may contain
characters that are not legal identifiers.
Change-Id: I62b8fe228a434a2040fd4ce47d220fc4d3398b41
Tested: on Linux.
This would cause double definition linker errors when included in
multiple compilation units.
Change-Id: Ie6fd4af018055a099343182a92a7776f2fea4725
Tested: on Linux.
Also fixed flatc not outputting these identifiers for files
compiled on the command-line.
Bug: 16983987
Change-Id: I8b714cfea3a8e144fa52133f62b2f7eda6eb044a
Tested: on Linux
Previously they were only generated for the root_type, making it
impossible to use the other types in the file as the root of a buffer.
Bug: 17206174
Change-Id: Ie71bed42ac3b22dcceae6385cbd5846c37e5f1b8
Tested: on Linux
Theoretically, an attacker could construct a FlatBuffer with the
sole purpose of making verification really expensive, essentially
DOS-ing a server that uses verification on FlatBuffers. This adds
a max table depth and max table amount at which point the
verifier declares the buffer malformed.
Bug: 16301336
Change-Id: I6b098c31d030d24c19e852b33609110658e66aa9
Tested: on OS X
This happens when the schema is parsed, to avoid compile time errors
later, which would be harder to understand.
Bug: 16325216
Change-Id: I24cabf1adaf1700796b91e3a9641bca43a68bfbd
Tested: on OS X.
Also cleaned up the general mkdir functionality.
Tested: on OS X.
Tested that a command line like: ./flatc -j -o foo/bar/baz tests/monster_test.fbs
generates files in foo/bar/baz/MyGame/Example/*.java which previously didn't exist.
Windows code was previously tested but has been moved, so needs to be tested again.
Change-Id: Iee943121b3be4f92961a3ec94c2cb578165f114a
Previously, it would ignore the fact that the type comes from a
different namespace. Now they are pre-declared in their own namespace,
and referenced with a qualified name if necessary.
Bug: 16851682
Change-Id: I5cb625b86d28e7436b9e93c70a0fa16a600d9884
Tested: on Linux
Please make sure you include the names of the affected language(s), compiler version(s), operating system version(s), and FlatBuffers version(s) in your issue title.
This helps us get the correct maintainers to look at your issue. Here are examples of good titles:
- Crash when accessing FlatBuffer [C++, gcc 4.8, OS X, master]
- Flatc converts a protobuf 'bytes' field to 'string' in fbs schema file [all languages, FlatBuffers 1.4]
"FlatBuffers is an efficient cross platform serialization library for C++, with support for Java, C# and Go. It was created at Google specifically for game development and other performance-critical applications.")
<divclass="textblock"><p>FlatBuffers is an efficient cross platform serialization library for C++, with support for Java and Go. It was created at Google specifically for game development and other performance-critical applications.</p>
<p>It is available as open source under the Apache license, v2 (see LICENSE.txt).</p>
<h2>Why use FlatBuffers?</h2>
<ul>
<li><b>Access to serialized data without parsing/unpacking</b> - What sets FlatBuffers apart is that it represents hierarchical data in a flat binary buffer in such a way that it can still be accessed directly without parsing/unpacking, while also still supporting data structure evolution (forwards/backwards compatibility).</li>
<li><b>Memory efficiency and speed</b> - The only memory needed to access your data is that of the buffer. It requires 0 additional allocations. FlatBuffers is also very suitable for use with mmap (or streaming), requiring only part of the buffer to be in memory. Access is close to the speed of raw struct access with only one extra indirection (a kind of vtable) to allow for format evolution and optional fields. It is aimed at projects where spending time and space (many memory allocations) to be able to access or construct serialized data is undesirable, such as in games or any other performance sensitive applications. See the <ahref="md__benchmarks.html">benchmarks</a> for details.</li>
<li><b>Flexible</b> - Optional fields means not only do you get great forwards and backwards compatibility (increasingly important for long-lived games: don't have to update all data with each new version!). It also means you have a lot of choice in what data you write and what data you don't, and how you design data structures.</li>
<li><b>Tiny code footprint</b> - Small amounts of generated code, and just a single small header as the minimum dependency, which is very easy to integrate. Again, see the benchmark section for details.</li>
<li><b>Strongly typed</b> - Errors happen at compile time rather than manually having to write repetitive and error prone run-time checks. Useful code can be generated for you.</li>
<li><pclass="startli"><b>Convenient to use</b> - Generated C++ code allows for terse access & construction code. Then there's optional functionality for parsing schemas and JSON-like text representations at runtime efficiently if needed (faster and more memory efficient than other JSON parsers).</p>
<pclass="startli">Java and Go code supports object-reuse.</p>
</li>
<li><b>Cross platform C++11/Java/Go code with no dependencies</b> - will work with any recent gcc/clang and VS2010. Comes with build files for the tests & samples (Android .mk files, and cmake for all other platforms).</li>
</ul>
<h3>Why not use Protocol Buffers, or .. ?</h3>
<p>Protocol Buffers is indeed relatively similar to FlatBuffers, with the primary difference being that FlatBuffers does not need a parsing/ unpacking step to a secondary representation before you can access data, often coupled with per-object memory allocation. The code is an order of magnitude bigger, too. Protocol Buffers has neither optional text import/export nor schema language features like unions.</p>
<h3>But all the cool kids use JSON!</h3>
<p>JSON is very readable (which is why we use it as our optional text format) and very convenient when used together with dynamically typed languages (such as JavaScript). When serializing data from statically typed languages, however, JSON not only has the obvious drawback of runtime inefficiency, but also forces you to write <em>more</em> code to access data (counterintuitively) due to its dynamic-typing serialization system. In this context, it is only a better choice for systems that have very little to no information ahead of time about what data needs to be stored.</p>
<p>Read more about the "why" of FlatBuffers in the <ahref="md__white_paper.html">white paper</a>.</p>
<h2>Usage in brief</h2>
<p>This section is a quick rundown of how to use this system. Subsequent sections provide a more in-depth usage guide.</p>
<ul>
<li>Write a schema file that allows you to define the data structures you may want to serialize. Fields can have a scalar type (ints/floats of all sizes), or they can be a: string; array of any type; reference to yet another object; or, a set of possible objects (unions). Fields are optional and have defaults, so they don't need to be present for every object instance.</li>
<li>Use <code>flatc</code> (the FlatBuffer compiler) to generate a C++ header (or Java/Go classes) with helper classes to access and construct serialized data. This header (say <code>mydata_generated.h</code>) only depends on <code>flatbuffers.h</code>, which defines the core functionality.</li>
<li>Use the <code>FlatBufferBuilder</code> class to construct a flat binary buffer. The generated functions allow you to add objects to this buffer recursively, often as simply as making a single function call.</li>
<li>Store or send your buffer somewhere!</li>
<li>When reading it back, you can obtain the pointer to the root object from the binary buffer, and from there traverse it conveniently in-place with <code>object->field()</code>.</li>
</ul>
<h2>In-depth documentation</h2>
<ul>
<li>How to <ahref="md__building.html">build the compiler</a> and samples on various platforms.</li>
<li>How to <ahref="md__compiler.html">use the compiler</a>.</li>
<li>How to <ahref="md__schemas.html">write a schema</a>.</li>
<li>How to <ahref="md__cpp_usage.html">use the generated C++ code</a> in your own programs.</li>
<li>How to <ahref="md__java_usage.html">use the generated Java code</a> in your own programs.</li>
<li>How to <ahref="md__go_usage.html">use the generated Go code</a> in your own programs.</li>
<li>Some <ahref="md__benchmarks.html">benchmarks</a> showing the advantage of using FlatBuffers.</li>
<li>A <ahref="md__white_paper.html">white paper</a> explaining the "why" of FlatBuffers.</li>
<li>A description of the <ahref="md__internals.html">internals</a> of FlatBuffers.</li>
<li>A formal <ahref="md__grammar.html">grammar</a> of the schema language.</li>
<divclass="textblock"><p>Comparing against other serialization solutions, running on Windows 7 64bit. We use the LITE runtime for Protocol Buffers (less code / lower overhead), Rapid JSON (one of the fastest C++ JSON parsers around), and pugixml, also one of the fastest XML parsers.</p>
<p>We compare against Flatbuffers with the binary wire format (as intended), and also with JSON as the wire format with the optional JSON parser (which, using a schema, parses JSON into a binary buffer that can then be accessed as before).</p>
<p>The benchmark object is a set of about 10 objects containing an array, 4 strings, and a large variety of int/float scalar values of all sizes, meant to be representative of game data, e.g. a scene format.</p>
<h3>Some other serialization systems we compared against but did not benchmark (yet), in rough order of applicability:</h3>
<ul>
<li>Cap'n'Proto promises to reduce Protocol Buffers much like FlatBuffers does, though with a more complicated binary encoding and less flexibility (no optional fields to allow deprecating fields or serializing with missing fields for which defaults exist). It currently also isn't fully cross-platform portable (lack of VS support).</li>
<li>msgpack: has very minimal forwards/backwards compatability support when used with the typed C++ interface. Also lacks VS2010 support.</li>
<li>Thrift: very similar to Protocol Buffers, but appears to be less efficient, and have more dependencies.</li>
<li>YAML: a superset of JSON and otherwise very similar. Used by e.g. Unity.</li>
<li>C# comes with built-in serialization functionality, as used by Unity also. Being tied to the language, and having no automatic versioning support limits its applicability.</li>
<li>Project Anarchy (the free mobile engine by Havok) comes with a serialization system, that however does no automatic versioning (have to code around new fields manually), is very much tied to the rest of the engine, and works without a schema to generate code (tied to your C++ class definition). </li>
<divclass="textblock"><p>There are project files for Visual Studio and Xcode that should allow you to build the compiler <code>flatc</code>, the samples and the tests out of the box.</p>
<p>Alternatively, the distribution comes with a <code>cmake</code> file that should allow you to build project/make files for any platform. For details on <code>cmake</code>, see <ahref="http://www.cmake.org">http://www.cmake.org</a>. In brief, depending on your platform, use one of e.g.: </p><preclass="fragment">cmake -G "Unix Makefiles"
cmake -G "Visual Studio 10"
cmake -G "Xcode"
</pre><p>Then, build as normal for your platform. This should result in a <code>flatc</code> executable, essential for the next steps. Note that to use clang instead of gcc, you may need to set up your environment variables, e.g. <code>CC=/usr/bin/clang CXX=/usr/bin/clang++ cmake -G "Unix Makefiles"</code>.</p>
<p>Optionally, run the <code>flattests</code> executable to ensure everything is working correctly on your system. If this fails, please contact us!</p>
<p>Note that you MUST be in the root of the FlatBuffers distribution when you run 'flattests' (and the samples), or it will fail to load its files.</p>
<p>Building should also produce two sample executables, <code>sample_binary</code> and <code>sample_text</code>, see the corresponding <code>.cpp</code> file in the samples directory.</p>
<p>There is an <code>android</code> directory that contains all you need to build the test executable on android (use the included <code>build_apk.sh</code> script, or use <code>ndk_build</code> / <code>adb</code> etc. as usual). Upon running, it will output to the log if tests succeeded or not.</p>
<p>There is usually no runtime to compile, as the code consists of a single header, <code>include/flatbuffers/flatbuffers.h</code>. You should add the <code>include</code> folder to your include paths. If you wish to be able to load schemas and/or parse text into binary buffers at runtime, you additionally need the other headers in <code>include/flatbuffers</code>. You must also compile/link <code>src/idl_parser.cpp</code> (and <code>src/idl_gen_text.cpp</code> if you also want to be able convert binary to text).</p>
<p>For applications on Google Play that integrate this library, usage is tracked. This tracking is done automatically using the embedded version string (flatbuffer_version_string), and helps us continue to optimize it. Aside from consuming a few extra bytes in your application binary, it shouldn't affect your application at all. We use this information to let us know if FlatBuffers is useful and if we should continue to invest in it. Since this is open source, you are free to remove the version string but we would appreciate if you would leave it in. </p>
</pre><p>The files are read and parsed in order, and can contain either schemas or data (see below). Later files can make use of definitions in earlier files.</p>
<p><code>--</code> indicates that the following files are binary files in FlatBuffer format conforming to the schema(s) indicated before it. Incompatible binary files currently will give unpredictable results (!)</p>
<p>Depending on the flags passed, additional files may be generated for each file processed:</p>
<ul>
<li><code>-c</code> : Generate a C++ header for all definitions in this file (as <code>filename_generated.h</code>). Skipped for data.</li>
<li><code>-j</code> : Generate Java classes. Skipped for data.</li>
<li><code>-b</code> : If data is contained in this file, generate a <code>filename.bin</code> containing the binary flatbuffer.</li>
<li><code>-t</code> : If data is contained in this file, generate a <code>filename.json</code> representing the data in the flatbuffer.</li>
<li><code>-o PATH</code> : Output all generated files to PATH (either absolute, or relative to the current directory). If omitted, PATH will be the current directory. PATH should end in your systems path separator, e.g. <code>/</code> or <code>\</code>.</li>
<li><code>-S</code> : Generate strict JSON (field names are enclosed in quotes). By default, no quotes are generated. </li>
<divclass="textblock"><p>Assuming you have written a schema using the above language in say <code>mygame.fbs</code> (FlatBuffer Schema, though the extension doesn't matter), you've generated a C++ header called <code>mygame_generated.h</code> using the compiler (e.g. <code>flatc -c mygame.fbs</code>), you can now start using this in your program by including the header. As noted, this header relies on <code>flatbuffers/flatbuffers.h</code>, which should be in your include path.</p>
<h3>Writing in C++</h3>
<p>To start creating a buffer, create an instance of <code>FlatBufferBuilder</code> which will contain the buffer as it grows: </p><preclass="fragment">FlatBufferBuilder fbb;
</pre><p>Before we serialize a Monster, we need to first serialize any objects that are contained there-in, i.e. we serialize the data tree using depth first, pre-order traversal. This is generally easy to do on any tree structures. For example: </p><preclass="fragment">auto name = fbb.CreateString("MyMonster");
</pre><p><code>CreateString</code> and <code>CreateVector</code> serialize these two built-in datatypes, and return offsets into the serialized data indicating where they are stored, such that <code>Monster</code> below can refer to them.</p>
<p><code>CreateString</code> can also take an <code>std::string</code>, or a <code>const char *</code> with an explicit length, and is suitable for holding UTF-8 and binary data if needed.</p>
<p><code>CreateVector</code> can also take an <code>std::vector</code>. The offset it returns is typed, i.e. can only be used to set fields of the correct type below. To create a vector of struct objects (which will be stored as contiguous memory in the buffer, use <code>CreateVectorOfStructs</code> instead. </p><preclass="fragment">Vec3 vec(1, 2, 3);
</pre><p><code>Vec3</code> is the first example of code from our generated header. Structs (unlike tables) translate to simple structs in C++, so we can construct them in a familiar way.</p>
<p>We have now serialized the non-scalar components of of the monster example, so we could create the monster something like this: </p><preclass="fragment">auto mloc = CreateMonster(fbb, &vec, 150, 80, name, inventory, Color_Red, 0, Any_NONE);
</pre><p>Note that we're passing <code>150</code> for the <code>mana</code> field, which happens to be the default value: this means the field will not actually be written to the buffer, since we'll get that value anyway when we query it. This is a nice space savings, since it is very common for fields to be at their default. It means we also don't need to be scared to add fields only used in a minority of cases, since they won't bloat up the buffer sizes if they're not actually used.</p>
<p>We do something similarly for the union field <code>test</code> by specifying a <code>0</code> offset and the <code>NONE</code> enum value (part of every union) to indicate we don't actually want to write this field. You can use <code>0</code> also as a default for other non-scalar types, such as strings, vectors and tables.</p>
<p>Tables (like <code>Monster</code>) give you full flexibility on what fields you write (unlike <code>Vec3</code>, which always has all fields set because it is a <code>struct</code>). If you want even more control over this (i.e. skip fields even when they are not default), instead of the convenient <code>CreateMonster</code> call we can also build the object field-by-field manually: </p><preclass="fragment">MonsterBuilder mb(fbb);
mb.add_pos(&vec);
mb.add_hp(80);
mb.add_name(name);
mb.add_inventory(inventory);
auto mloc = mb.Finish();
</pre><p>We start with a temporary helper class <code>MonsterBuilder</code> (which is defined in our generated code also), then call the various <code>add_</code> methods to set fields, and <code>Finish</code> to complete the object. This is pretty much the same code as you find inside <code>CreateMonster</code>, except we're leaving out a few fields. Fields may also be added in any order, though orderings with fields of the same size adjacent to each other most efficient in size, due to alignment. You should not nest these Builder classes (serialize your data in pre-order).</p>
<p>Regardless of whether you used <code>CreateMonster</code> or <code>MonsterBuilder</code>, you now have an offset to the root of your data, and you can finish the buffer using: </p><preclass="fragment">FinishMonsterBuffer(fbb, mloc);
</pre><p>The buffer is now ready to be stored somewhere, sent over the network, be compressed, or whatever you'd like to do with it. You can access the start of the buffer with <code>fbb.GetBufferPointer()</code>, and it's size from <code>fbb.GetSize()</code>.</p>
<p><code>samples/sample_binary.cpp</code> is a complete code sample similar to the code above, that also includes the reading code below.</p>
<h3>Reading in C++</h3>
<p>If you've received a buffer from somewhere (disk, network, etc.) you can directly start traversing it using: </p><preclass="fragment">auto monster = GetMonster(buffer_pointer);
</pre><p><code>monster</code> is of type <code>Monster *</code>, and points to somewhere inside your buffer. If you look in your generated header, you'll see it has convenient accessors for all fields, e.g. </p><preclass="fragment">assert(monster->hp() == 80);
</pre><p>These should all be true. Note that we never stored a <code>mana</code> value, so it will return the default.</p>
<p>To access sub-objects, in this case the <code>Vec3</code>: </p><preclass="fragment">auto pos = monster->pos();
assert(pos);
assert(pos->z() == 3);
</pre><p>If we had not set the <code>pos</code> field during serialization, it would be <code>NULL</code>.</p>
<p>Similarly, we can access elements of the inventory array: </p><preclass="fragment">auto inv = monster->inventory();
assert(inv);
assert(inv->Get(9) == 9);
</pre><h3>Direct memory access</h3>
<p>As you can see from the above examples, all elements in a buffer are accessed through generated accessors. This is because everything is stored in little endian format on all platforms (the accessor performs a swap operation on big endian machines), and also because the layout of things is generally not known to the user.</p>
<p>For structs, layout is deterministic and guaranteed to be the same accross platforms (scalars are aligned to their own size, and structs themselves to their largest member), and you are allowed to access this memory directly by using <code>sizeof()</code> and <code>memcpy</code> on the pointer to a struct, or even an array of structs.</p>
<p>To compute offsets to sub-elements of a struct, make sure they are a structs themselves, as then you can use the pointers to figure out the offset without having to hardcode it. This is handy for use of arrays of structs with calls like <code>glVertexAttribPointer</code> in OpenGL or similar APIs.</p>
<p>It is important to note is that structs are still little endian on all machines, so only use tricks like this if you can guarantee you're not shipping on a big endian machine (an <code>assert(FLATBUFFERS_LITTLEENDIAN)</code> would be wise).</p>
<h3>Access of untrusted buffers</h3>
<p>The generated accessor functions access fields over offsets, which is very quick. These offsets are not verified at run-time, so a malformed buffer could cause a program to crash by accessing random memory.</p>
<p>When you're processing large amounts of data from a source you know (e.g. your own generated data on disk), this is acceptable, but when reading data from the network that can potentially have been modified by an attacker, this is undesirable.</p>
<p>For this reason, you can optionally use a buffer verifier before you access the data. This verifier will check all offsets, all sizes of fields, and null termination of strings to ensure that when a buffer is accessed, all reads will end up inside the buffer.</p>
<p>Each root type will have a verification function generated for it, e.g. for <code>Monster</code>, you can call:</p>
<p>bool ok = VerifyMonsterBuffer(Verifier(buf, len));</p>
<p>if <code>ok</code> is true, the buffer is safe to read.</p>
<p>Besides untrusted data, this function may be useful to call in debug mode, as extra insurance against data being corrupted somewhere along the way.</p>
<p>While verifying a buffer isn't "free", it is typically faster than a full traversal (since any scalar data is not actually touched), and since it may cause the buffer to be brought into cache before reading, the actual overhead may be even lower than expected.</p>
<h2>Text & schema parsing</h2>
<p>Using binary buffers with the generated header provides a super low overhead use of FlatBuffer data. There are, however, times when you want to use text formats, for example because it interacts better with source control, or you want to give your users easy access to data.</p>
<p>Another reason might be that you already have a lot of data in JSON format, or a tool that generates JSON, and if you can write a schema for it, this will provide you an easy way to use that data directly.</p>
<p>(see the schema documentation for some specifics on the JSON format accepted).</p>
<p>There are two ways to use text formats:</p>
<h3>Using the compiler as a conversion tool</h3>
<p>This is the preferred path, as it doesn't require you to add any new code to your program, and is maximally efficient since you can ship with binary data. The disadvantage is that it is an extra step for your users/developers to perform, though you might be able to automate it. </p><preclass="fragment">flatc -b myschema.fbs mydata.json
</pre><p>This will generate the binary file <code>mydata_wire.bin</code> which can be loaded as before.</p>
<h3>Making your program capable of loading text directly</h3>
<p>This gives you maximum flexibility. You could even opt to support both, i.e. check for both files, and regenerate the binary from text when required, otherwise just load the binary.</p>
<p>This option is currently only available for C++, or Java through JNI.</p>
<p>As mentioned in the section "Building" above, this technique requires you to link a few more files into your program, and you'll want to include <code>flatbuffers/idl.h</code>.</p>
<p>Load text (either a schema or json) into an in-memory buffer (there is a convenient <code>LoadFile()</code> utility function in <code>flatbuffers/util.h</code> if you wish). Construct a parser: </p><preclass="fragment">flatbuffers::Parser parser;
</pre><p>Now you can parse any number of text files in sequence: </p><preclass="fragment">parser.Parse(text_file.c_str());
</pre><p>This works similarly to how the command-line compiler works: a sequence of files parsed by the same <code>Parser</code> object allow later files to reference definitions in earlier files. Typically this means you first load a schema file (which populates <code>Parser</code> with definitions), followed by one or more JSON files.</p>
<p>If there were any parsing errors, <code>Parse</code> will return <code>false</code>, and <code>Parser::err</code> contains a human readable error string with a line number etc, which you should present to the creator of that file.</p>
<p>After each JSON file, the <code>Parser::fbb</code> member variable is the <code>FlatBufferBuilder</code> that contains the binary buffer version of that file, that you can access as described above.</p>
<p><code>samples/sample_text.cpp</code> is a code sample showing the above operations.</p>
<h3>Threading</h3>
<p>None of the code is thread-safe, by design. That said, since currently a FlatBuffer is read-only and entirely <code>const</code>, reading by multiple threads is possible. </p>
<divclass="textblock"><p>There's experimental support for reading FlatBuffers in Go. Generate code for Go with the <code>-g</code> option to <code>flatc</code>.</p>
<p>See <code>go_test.go</code> for an example. You import the generated code, read a FlatBuffer binary file into a <code>[]byte</code>, which you pass to the <code>GetRootAsMonster</code> function: </p><preclass="fragment">import (
example "MyGame/Example"
flatbuffers "github.com/google/flatbuffers/go"
io/ioutil
)
buf, err := ioutil.ReadFile("monster.dat")
// handle err
monster := example.GetRootAsMonster(buf, 0)
</pre><p>Now you can access values like this: </p><preclass="fragment">hp := monster.Hp()
pos := monster.Pos(nil)
</pre><p>Note that whenever you access a new object like in the <code>Pos</code> example above, a new temporary accessor object gets created. If your code is very performance sensitive (you iterate through a lot of objects), you can replace nil with a pointer to a <code>Vec3</code> object you've already created. This allows you to reuse it across many calls and reduce the amount of object allocation (and thus garbage collection) your program does.</p>
<p>To access vectors you pass an extra index to the vector field accessor. Then a second method with the same name suffixed by <code>Length</code> let's you know the number of elements you can access: </p><preclass="fragment">for i := 0; i < monster.InventoryLength(); i++ {
monster.Inventory(i) // do something here
}
</pre><p>You can also construct these buffers in Go using the functions found in the generated code, and the FlatBufferBuilder class: </p><preclass="fragment">builder := flatbuffers.NewBuilder(0)
</pre><p>Unlike C++, Go does not support table creation functions like 'createMonster()'. This is to create the buffer without using temporary object allocation (since the <code>Vec3</code> is an inline component of <code>Monster</code>, it has to be created right where it is added, whereas the name and the inventory are not inline). Structs do have convenient methods that allow you to construct them in one call. These also have arguments for nested structs, e.g. if a struct has a field <code>a</code> and a nested struct field <code>b</code> (which has fields <code>c</code> and <code>d</code>), then the arguments will be <code>a</code>, <code>c</code> and <code>d</code>.</p>
<p>Vectors also use this start/end pattern to allow vectors of both scalar types and structs: </p><preclass="fragment">example.MonsterStartInventoryVector(builder, 5)
for i := 4; i >= 0; i-- {
builder.PrependByte(byte(i))
}
inv := builder.EndVector(5)
</pre><p>The generated method 'StartInventoryVector' is provided as a convenience function which calls 'StartVector' with the correct element size of the vector type which in this case is 'ubyte' or 1 byte per vector element. You pass the number of elements you want to write. You write the elements backwards since the buffer is being constructed back to front.</p>
<p>There are <code>Prepend</code> functions for all the scalar types. You use <code>PrependUOffset</code> for any previously constructed objects (such as other tables, strings, vectors). For structs, you use the appropriate <code>create</code> function in-line, as shown above in the <code>Monster</code> example.</p>
<h2>Text Parsing</h2>
<p>There currently is no support for parsing text (Schema's and JSON) directly from Go, though you could use the C++ parser through cgo. Please see the C++ documentation for more on text parsing. </p>
<divclass="textblock"><p>This section is entirely optional for the use of FlatBuffers. In normal usage, you should never need the information contained herein. If you're interested however, it should give you more of an appreciation of why FlatBuffers is both efficient and convenient.</p>
<h3>Format components</h3>
<p>A FlatBuffer is a binary file and in-memory format consisting mostly of scalars of various sizes, all aligned to their own size. Each scalar is also always represented in little-endian format, as this corresponds to all commonly used CPUs today. FlatBuffers will also work on big-endian machines, but will be slightly slower because of additional byte-swap intrinsics.</p>
<p>On purpose, the format leaves a lot of details about where exactly things live in memory undefined, e.g. fields in a table can have any order, and objects to some extend can be stored in many orders. This is because the format doesn't need this information to be efficient, and it leaves room for optimization and extension (for example, fields can be packed in a way that is most compact). Instead, the format is defined in terms of offsets and adjacency only. This may mean two different implementations may produce different binaries given the same input values, and this is perfectly valid.</p>
<h3>Format identification</h3>
<p>The format also doesn't contain information for format identification and versioning, which is also by design. FlatBuffers is a statically typed system, meaning the user of a buffer needs to know what kind of buffer it is. FlatBuffers can of course be wrapped inside other containers where needed, or you can use its union feature to dynamically identify multiple possible sub-objects stored. Additionally, it can be used together with the schema parser if full reflective capabilities are desired.</p>
<p>Versioning is something that is intrinsically part of the format (the optionality / extensibility of fields), so the format itself does not need a version number (it's a meta-format, in a sense). We're hoping that this format can accommodate all data needed. If format breaking changes are ever necessary, it would become a new kind of format rather than just a variation.</p>
<h3>Offsets</h3>
<p>The most important and generic offset type (see <code>flatbuffers.h</code>) is <code>uoffset_t</code>, which is currently always a <code>uint32_t</code>, and is used to refer to all tables/unions/strings/vectors (these are never stored in-line). 32bit is intentional, since we want to keep the format binary compatible between 32 and 64bit systems, and a 64bit offset would bloat the size for almost all uses. A version of this format with 64bit (or 16bit) offsets is easy to set when needed. Unsigned means they can only point in one direction, which typically is forward (towards a higher memory location). Any backwards offsets will be explicitly marked as such.</p>
<p>The format starts with an <code>uoffset_t</code> to the root object in the buffer.</p>
<p>We have two kinds of objects, structs and tables.</p>
<h3>Structs</h3>
<p>These are the simplest, and as mentioned, intended for simple data that benefits from being extra efficient and doesn't need versioning / extensibility. They are always stored inline in their parent (a struct, table, or vector) for maximum compactness. Structs define a consistent memory layout where all components are aligned to their size, and structs aligned to their largest scalar member. This is done independent of the alignment rules of the underlying compiler to guarantee a cross platform compatible layout. This layout is then enforced in the generated code.</p>
<h3>Tables</h3>
<p>These start with an <code>soffset_t</code> to a vtable (signed version of <code>uoffset_t</code>, since vtables may be stored anywhere), followed by all the fields as aligned scalars (or offsets). Unlike structs, not all fields need to be present. There is no set order and layout.</p>
<p>To be able to access fields regardless of these uncertainties, we go through a vtable of offsets. Vtables are shared between any objects that happen to have the same vtable values.</p>
<p>The elements of a vtable are all of type <code>voffset_t</code>, which is a <code>uint16_t</code>. The first element is the number of elements of the vtable, including this one. The second one is the size of the object, in bytes (including the vtable offset). This size is used for streaming, to know how many bytes to read to be able to access all fields of the object. The remaining elements are the N offsets, where N is the amount of fields declared in the schema when the code that constructed this buffer was compiled (thus, the size of the table is N + 2).</p>
<p>All accessor functions in the generated code for tables contain the offset into this table as a constant. This offset is checked against the first field (the number of elements), to protect against newer code reading older data. If this offset is out of range, or the vtable entry is 0, that means the field is not present in this object, and the default value is return. Otherwise, the entry is used as offset to the field to be read.</p>
<h3>Strings and Vectors</h3>
<p>Strings are simply a vector of bytes, and are always null-terminated. Vectors are stored as contiguous aligned scalar elements prefixed by a 32bit element count (not including any null termination).</p>
<h3>Construction</h3>
<p>The current implementation constructs these buffers backwards (starting at the highest memory address of the buffer), since that significantly reduces the amount of bookkeeping and simplifies the construction API.</p>
<h3>Code example</h3>
<p>Here's an example of the code that gets generated for the <code>samples/monster.fbs</code>. What follows is the entire file, broken up by comments: </p><preclass="fragment">// automatically generated, do not modify
</pre><p>These ugly macros do a couple of things: they turn off any padding the compiler might normally do, since we add padding manually (though none in this example), and they enforce alignment chosen by FlatBuffers. This ensures the layout of this struct will look the same regardless of compiler and platform. Note that the fields are private: this is because these store little endian scalars regardless of platform (since this is part of the serialized data). <code>EndianScalar</code> then converts back and forth, which is a no-op on all current mobile and desktop platforms, and a single machine instruction on the few remaining big endian platforms. </p><preclass="fragment">struct Monster : private flatbuffers::Table {
</pre><p>Tables are a bit more complicated. A table accessor struct is used to point at the serialized data for a table, which always starts with an offset to its vtable. It derives from <code>Table</code>, which contains the <code>GetField</code> helper functions. GetField takes a vtable offset, and a default value. It will look in the vtable at that offset. If the offset is out of bounds (data from an older version) or the vtable entry is 0, the field is not present and the default is returned. Otherwise, it uses the entry as an offset into the table to locate the field. </p><preclass="fragment">struct MonsterBuilder {
</pre><p><code>MonsterBuilder</code> is the base helper struct to construct a table using a <code>FlatBufferBuilder</code>. You can add the fields in any order, and the <code>Finish</code> call will ensure the correct vtable gets generated. </p><preclass="fragment">inline flatbuffers::Offset<Monster> CreateMonster(flatbuffers::FlatBufferBuilder &_fbb, const Vec3 *pos, int16_t mana, int16_t hp, flatbuffers::Offset<flatbuffers::String> name, flatbuffers::Offset<flatbuffers::Vector<uint8_t>> inventory, int8_t color) {
MonsterBuilder builder_(_fbb);
builder_.add_inventory(inventory);
builder_.add_name(name);
builder_.add_pos(pos);
builder_.add_hp(hp);
builder_.add_mana(mana);
builder_.add_color(color);
return builder_.Finish();
}
</pre><p><code>CreateMonster</code> is a convenience function that calls all functions in <code>MonsterBuilder</code> above for you. Note that if you pass values which are defaults as arguments, it will not actually construct that field, so you can probably use this function instead of the builder class in almost all cases. </p><preclass="fragment">inline const Monster *GetMonster(const void *buf) { return flatbuffers::GetRoot<Monster>(buf); }
</pre><p>This function is only generated for the root table type, to be able to start traversing a FlatBuffer from a raw buffer pointer. </p><preclass="fragment">}; // namespace MyGame
<divclass="textblock"><p>FlatBuffers supports reading and writing binary FlatBuffers in Java. Generate code for Java with the <code>-j</code> option to <code>flatc</code>.</p>
<p>See <code>javaTest.java</code> for an example. Essentially, you read a FlatBuffer binary file into a <code>byte[]</code>, which you then turn into a <code>ByteBuffer</code>, which you pass to the <code>getRootAsMyRootType</code> function: </p><preclass="fragment">ByteBuffer bb = ByteBuffer.wrap(data);
Monster monster = Monster.getRootAsMonster(bb);
</pre><p>Now you can access values much like C++: </p><preclass="fragment">short hp = monster.hp();
Vec3 pos = monster.pos();
</pre><p>Note that whenever you access a new object like in the <code>pos</code> example above, a new temporary accessor object gets created. If your code is very performance sensitive (you iterate through a lot of objects), there's a second <code>pos()</code> method to which you can pass a <code>Vec3</code> object you've already created. This allows you to reuse it across many calls and reduce the amount of object allocation (and thus garbage collection) your program does.</p>
<p>Java does not support unsigned scalars. This means that any unsigned types you use in your schema will actually be represented as a signed value. This means all bits are still present, but may represent a negative value when used. For example, to read a <code>byte b</code> as an unsigned number, you can do: <code>(short)(b & 0xFF)</code></p>
<p>Sadly the string accessors currently always create a new string when accessed, since FlatBuffer's UTF-8 strings can't be read in-place by Java.</p>
<p>Vector access is also a bit different from C++: you pass an extra index to the vector field accessor. Then a second method with the same name suffixed by <code>Length</code> let's you know the number of elements you can access: </p><preclass="fragment">for (int i = 0; i < monster.inventoryLength(); i++)
monster.inventory(i); // do something here
</pre><p>You can also construct these buffers in Java using the static methods found in the generated code, and the FlatBufferBuilder class: </p><preclass="fragment">FlatBufferBuilder fbb = new FlatBufferBuilder();
</pre><p>As you can see, the Java code for tables does not use a convenient <code>createMonster</code> call like the C++ code. This is to create the buffer without using temporary object allocation.</p>
<p>It's important to understand that fields that are structs are inline (like <code>Vec3</code> above), and MUST thus be created between the start and end calls of a table. Everything else (other tables, strings, vectors) MUST be created before the start of the table they are referenced in.</p>
<p>Structs do have convenient methods that even have arguments for nested structs.</p>
<p>As you can see, references to other objects (e.g. the string above) are simple ints, and thus do not have the type-safety of the Offset type in C++. Extra case must thus be taken that you set the right offset on the right field.</p>
<p>Vectors also use this start/end pattern to allow vectors of both scalar types and structs: </p><preclass="fragment">Monster.startInventoryVector(fbb, 5);
for (byte i = 4; i >=0; i--) fbb.addByte(i);
int inv = fbb.endVector();
</pre><p>You can use the generated method <code>startInventoryVector</code> to conveniently call <code>startVector</code> with the right element size. You pass the number of elements you want to write. You write the elements backwards since the buffer is being constructed back to front.</p>
<p>There are <code>add</code> functions for all the scalar types. You use <code>addOffset</code> for any previously constructed objects (such as other tables, strings, vectors). For structs, you use the appropriate <code>create</code> function in-line, as shown above in the <code>Monster</code> example.</p>
<h2>Text Parsing</h2>
<p>There currently is no support for parsing text (Schema's and JSON) directly from Java, though you could use the C++ parser through JNI. Please see the C++ documentation for more on text parsing. </p>
<divclass="textblock"><p>The syntax of the schema language (aka IDL, Interface Definition Language) should look quite familiar to users of any of the C family of languages, and also to users of other IDLs. Let's look at an example first: </p><preclass="fragment">// example IDL file
namespace MyGame;
enum Color : byte { Red = 1, Green, Blue }
union Any { Monster, Weapon, Pickup }
struct Vec3 {
x:float;
y:float;
z:float;
}
table Monster {
pos:Vec3;
mana:short = 150;
hp:short = 100;
name:string;
friendly:bool = false (deprecated, priority: 1);
inventory:[ubyte];
color:Color = Blue;
test:Any;
}
root_type Monster;
</pre><p>(Weapon & Pickup not defined as part of this example).</p>
<h3>Tables</h3>
<p>Tables are the main way of defining objects in FlatBuffers, and consist of a name (here <code>Monster</code>) and a list of fields. Each field has a name, a type, and optionally a default value (if omitted, it defaults to 0 / NULL).</p>
<p>Each field is optional: It does not have to appear in the wire representation, and you can choose to omit fields for each individual object. As a result, you have the flexibility to add fields without fear of bloating your data. This design is also FlatBuffer's mechanism for forward and backwards compatibility. Note that:</p>
<ul>
<li>You can add new fields in the schema ONLY at the end of a table definition. Older data will still read correctly, and give you the default value when read. Older code will simply ignore the new field. If you want to have flexibility to use any order for fields in your schema, you can manually assign ids (much like Protocol Buffers), see the <code>id</code> attribute below.</li>
<li>You cannot delete fields you don't use anymore from the schema, but you can simply stop writing them into your data for almost the same effect. Additionally you can mark them as <code>deprecated</code> as in the example above, which will prevent the generation of accessors in the generated C++, as a way to enforce the field not being used any more. (careful: this may break code!).</li>
<li>You may change field names and table names, if you're ok with your code breaking until you've renamed them there too.</li>
</ul>
<h3>Structs</h3>
<p>Similar to a table, only now none of the fields are optional (so no defaults either), and fields may not be added or be deprecated. Structs may only contain scalars or other structs. Use this for simple objects where you are very sure no changes will ever be made (as quite clear in the example <code>Vec3</code>). Structs use less memory than tables and are even faster to access (they are always stored in-line in their parent object, and use no virtual table).</p>
<h3>Types</h3>
<p>Builtin scalar types are:</p>
<ul>
<li>8 bit: <code>byte ubyte bool</code></li>
<li>16 bit: <code>short ushort</code></li>
<li>32 bit: <code>int uint float</code></li>
<li>64 bit: <code>long ulong double</code></li>
<li>Vector of any other type (denoted with <code>[type]</code>). Nesting vectors is not supported, instead you can wrap the inner vector in a table.</li>
<li><code>string</code>, which may only hold UTF-8 or 7-bit ASCII. For other text encodings or general binary data use vectors (<code>[byte]</code> or <code>[ubyte]</code>) instead.</li>
<li>References to other tables or structs, enums or unions (see below).</li>
</ul>
<p>You can't change types of fields once they're used, with the exception of same-size data where a <code>reinterpret_cast</code> would give you a desirable result, e.g. you could change a <code>uint</code> to an <code>int</code> if no values in current data use the high bit yet.</p>
<h3>(Default) Values</h3>
<p>Values are a sequence of digits, optionally followed by a <code>.</code> and more digits for float constants, and optionally prefixed by a <code>-</code>. Non-scalar defaults are currently not supported (always NULL).</p>
<p>You generally do not want to change default values after they're initially defined. Fields that have the default value are not actually stored in the serialized data but are generated in code, so when you change the default, you'd now get a different value than from code generated from an older version of the schema. There are situations however where this may be desirable, especially if you can ensure a simultaneous rebuild of all code.</p>
<h3>Enums</h3>
<p>Define a sequence of named constants, each with a given value, or increasing by one from the previous one. The default first value is <code>0</code>. As you can see in the enum declaration, you specify the underlying integral type of the enum with <code>:</code> (in this case <code>byte</code>), which then determines the type of any fields declared with this enum type.</p>
<h3>Unions</h3>
<p>Unions share a lot of properties with enums, but instead of new names for constants, you use names of tables. You can then declare a union field which can hold a reference to any of those types, and additionally a hidden field with the suffix <code>_type</code> is generated that holds the corresponding enum value, allowing you to know which type to cast to at runtime.</p>
<h3>Namespaces</h3>
<p>These will generate the corresponding namespace in C++ for all helper code, and packages in Java. You can use <code>.</code> to specify nested namespaces / packages.</p>
<h3>Root type</h3>
<p>This declares what you consider to be the root table (or struct) of the serialized data. This is particular important for parsing JSON data, which doesn't include object type information.</p>
<h3>File identification and extension</h3>
<p>Typically, a FlatBuffer binary buffer is not self-describing, i.e. it needs you to know its schema to parse it correctly. But if you want to use a FlatBuffer as a file format, it would be convenient to be able to have a "magic number" in there, like most file formats have, to be able to do a sanity check to see if you're reading the kind of file you're expecting.</p>
<p>Now, you can always prefix a FlatBuffer with your own file header, but FlatBuffers has a built-in way to add an identifier to a FlatBuffer that takes up minimal space, and keeps the buffer compatible with buffers that don't have such an identifier.</p>
<p>You can specify in a schema, similar to <code>root_type</code>, that you intend for this type of FlatBuffer to be used as a file format: </p><preclass="fragment">file_identifier "MYFI";
</pre><p>Identifiers must always be exactly 4 characters long. These 4 characters will end up as bytes at offsets 4-7 (inclusive) in the buffer.</p>
<p>For any schema that has such an identifier, <code>flatc</code> will automatically add the identifier to any binaries it generates (with <code>-b</code>), and generated calls like <code>FinishMonsterBuffer</code> also add the identifier. If you have specified an identifier and wish to generate a buffer without one, you can always still do so by calling <code>FlatBufferBuilder::Finish</code> explicitly.</p>
<p>After loading a buffer, you can use a call like <code>MonsterBufferHasIdentifier</code> to check if the identifier is present.</p>
<p>Additionally, by default <code>flatc</code> will output binary files as <code>.bin</code>. This declaration in the schema will change that to whatever you want: </p><preclass="fragment">file_extension "ext";
</pre><h3>Comments & documentation</h3>
<p>May be written as in most C-based languages. Additionally, a triple comment (<code>///</code>) on a line by itself signals that a comment is documentation for whatever is declared on the line after it (table/struct/field/enum/union/element), and the comment is output in the corresponding C++ code. Multiple such lines per item are allowed.</p>
<h3>Attributes</h3>
<p>Attributes may be attached to a declaration, behind a field, or after the name of a table/struct/enum/union. These may either have a value or not. Some attributes like <code>deprecated</code> are understood by the compiler, others are simply ignored (like <code>priority</code>), but are available to query if you parse the schema at runtime. This is useful if you write your own code generators/editors etc., and you wish to add additional information specific to your tool (such as a help text).</p>
<p>Current understood attributes:</p>
<ul>
<li><code>id: n</code> (on a table field): manually set the field identifier to <code>n</code>. If you use this attribute, you must use it on ALL fields of this table, and the numbers must be a contiguous range from 0 onwards. Additionally, since a union type effectively adds two fields, its id must be that of the second field (the first field is the type field and not explicitly declared in the schema). For example, if the last field before the union field had id 6, the union field should have id 8, and the unions type field will implicitly be 7. IDs allow the fields to be placed in any order in the schema. When a new field is added to the schema is must use the next available ID.</li>
<li><code>deprecated</code> (on a field): do not generate accessors for this field anymore, code should stop using this data.</li>
<li><code>original_order</code> (on a table): since elements in a table do not need to be stored in any particular order, they are often optimized for space by sorting them to size. This attribute stops that from happening.</li>
<li><code>force_align: size</code> (on a struct): force the alignment of this struct to be something higher than what it is naturally aligned to. Causes these structs to be aligned to that amount inside a buffer, IF that buffer is allocated with that alignment (which is not necessarily the case for buffers accessed directly inside a <code>FlatBufferBuilder</code>).</li>
<li><code>bit_flags</code> (on an enum): the values of this field indicate bits, meaning that any value N specified in the schema will end up representing 1<<N, or if you don't specify values at all, you'll get the sequence 1, 2, 4, 8, ...</li>
</ul>
<h2>JSON Parsing</h2>
<p>The same parser that parses the schema declarations above is also able to parse JSON objects that conform to this schema. So, unlike other JSON parsers, this parser is strongly typed, and parses directly into a FlatBuffer (see the compiler documentation on how to do this from the command line, or the C++ documentation on how to do this at runtime).</p>
<p>Besides needing a schema, there are a few other changes to how it parses JSON:</p>
<ul>
<li>It accepts field names with and without quotes, like many JSON parsers already do. It outputs them without quotes as well, though can be made to output them using the <code>strict_json</code> flag.</li>
<li>If a field has an enum type, the parser will recognize symbolic enum values (with or without quotes) instead of numbers, e.g. <code>field: EnumVal</code>. If a field is of integral type, you can still use symbolic names, but values need to be prefixed with their type and need to be quoted, e.g. <code>field: "Enum.EnumVal"</code>. For enums representing flags, you may place multiple inside a string separated by spaces to OR them, e.g. <code>field: "EnumVal1 EnumVal2"</code> or <code>field: "Enum.EnumVal1 Enum.EnumVal2"</code>.</li>
</ul>
<h2>Gotchas</h2>
<h3>Schemas and version control</h3>
<p>FlatBuffers relies on new field declarations being added at the end, and earlier declarations to not be removed, but be marked deprecated when needed. We think this is an improvement over the manual number assignment that happens in Protocol Buffers (and which is still an option using the <code>id</code> attribute mentioned above).</p>
<p>One place where this is possibly problematic however is source control. If user A adds a field, generates new binary data with this new schema, then tries to commit both to source control after user B already committed a new field also, and just auto-merges the schema, the binary files are now invalid compared to the new schema.</p>
<p>The solution of course is that you should not be generating binary data before your schema changes have been committed, ensuring consistency with the rest of the world. If this is not practical for you, use explicit field ids, which should always generate a merge conflict if two people try to allocate the same id. </p>
<divclass="title">FlatBuffers white paper </div></div>
</div><!--header-->
<divclass="contents">
<divclass="textblock"><p>This document tries to shed some light on to the "why" of FlatBuffers, a new serialization library.</p>
<h2>Motivation</h2>
<p>Back in the good old days, performance was all about instructions and cycles. Nowadays, processing units have run so far ahead of the memory subsystem, that making an efficient application should start and finish with thinking about memory. How much you use of it. How you lay it out and access it. How you allocate it. When you copy it.</p>
<p>Serialization is a pervasive activity in a lot programs, and a common source of memory inefficiency, with lots of temporary data structures needed to parse and represent data, and inefficient allocation patterns and locality.</p>
<p>If it would be possible to do serialization with no temporary objects, no additional allocation, no copying, and good locality, this could be of great value. The reason serialization systems usually don't manage this is because it goes counter to forwards/backwards compatability, and platform specifics like endianness and alignment.</p>
<p>FlatBuffers is what you get if you try anyway.</p>
<p>In particular, FlatBuffers focus is on mobile hardware (where memory size and memory bandwidth is even more constrained than on desktop hardware), and applications that have the highest performance needs: games.</p>
<h2>FlatBuffers</h2>
<p><em>This is a summary of FlatBuffers functionality, with some rationale. A more detailed description can be found in the FlatBuffers documentation.</em></p>
<h3>Summary</h3>
<p>A FlatBuffer is a binary buffer containing nested objects (structs, tables, vectors,..) organized using offsets so that the data can be traversed in-place just like any pointer-based data structure. Unlike most in-memory data structures however, it uses strict rules of alignment and endianness (always little) to ensure these buffers are cross platform. Additionally, for objects that are tables, FlatBuffers provides forwards/backwards compatibility and general optionality of fields, to support most forms of format evolution.</p>
<p>You define your object types in a schema, which can then be compiled to C++ or Java for low to zero overhead reading & writing. Optionally, JSON data can be dynamically parsed into buffers.</p>
<h3>Tables</h3>
<p>Tables are the cornerstone of FlatBuffers, since format evolution is essential for most applications of serialization. Typically, dealing with format changes is something that can be done transparently during the parsing process of most serialization solutions out there. But a FlatBuffer isn't parsed before it is accessed.</p>
<p>Tables get around this by using an extra indirection to access fields, through a <em>vtable</em>. Each table comes with a vtable (which may be shared between multiple tables with the same layout), and contains information where fields for this particular kind of instance of vtable are stored. The vtable may also indicate that the field is not present (because this FlatBuffer was written with an older version of the software, of simply because the information was not necessary for this instance, or deemed deprecated), in which case a default value is returned.</p>
<p>Tables have a low overhead in memory (since vtables are small and shared) and in access cost (an extra indirection), but provide great flexibility. Tables may even cost less memory than the equivalent struct, since fields do not need to be stored when they are equal to their default.</p>
<p>FlatBuffers additionally offers "naked" structs, which do not offer forwards/backwards compatibility, but can be even smaller (useful for very small objects that are unlikely to change, like e.g. a coordinate pair or a RGBA color).</p>
<h3>Schemas</h3>
<p>While schemas reduce some generality (you can't just read any data without having its schema), they have a lot of upsides:</p>
<ul>
<li>Most information about the format can be factored into the generated code, reducing memory needed to store data, and time to access it.</li>
<li>The strong typing of the data definitions means less error checking/handling at runtime (less can go wrong).</li>
<li>A schema enables us to access a buffer without parsing.</li>
</ul>
<p>FlatBuffer schemas are fairly similar to those of the incumbent, Protocol Buffers, and generally should be readable to those familiar with the C family of languages. We chose to improve upon the features offered by .proto files in the following ways:</p>
<ul>
<li>Deprecation of fields instead of manual field id assignment. Extending an object in a .proto means hunting for a free slot among the numbers (preferring lower numbers since they have a more compact representation). Besides being inconvenient, it also makes removing fields problematic: you either have to keep them, not making it obvious that this field shouldn't be read/written anymore, and still generating accessors. Or you remove it, but now you risk that there's still old data around that uses that field by the time someone reuses that field id, with nasty consequences.</li>
<li>Differentiating between tables and structs (see above). Effectively all table fields are <code>optional</code>, and all struct fields are <code>required</code>.</li>
<li>Having a native vector type instead of <code>repeated</code>. This gives you a length without having to collect all items, and in the case of scalars provides for a more compact representation, and one that guarantees adjacency.</li>
<li>Having a native <code>union</code> type instead of using a series of <code>optional</code> fields, all of which must be checked individually.</li>
<li>Being able to define defaults for all scalars, instead of having to deal with their optionality at each access.</li>
<li>A parser that can deal with both schemas and data definitions (JSON compatible) uniformly. </li>
<trid="row_1_"><tdclass="entry"><spanstyle="width:16px;display:inline-block;"> </span><aclass="el"href="md__compiler.html"target="_self">Using the schema compiler</a></td><tdclass="desc"></td></tr>
<trid="row_2_"class="even"><tdclass="entry"><spanstyle="width:16px;display:inline-block;"> </span><aclass="el"href="md__schemas.html"target="_self">Writing a schema</a></td><tdclass="desc"></td></tr>
<trid="row_3_"><tdclass="entry"><spanstyle="width:16px;display:inline-block;"> </span><aclass="el"href="md__cpp_usage.html"target="_self">Use in C++</a></td><tdclass="desc"></td></tr>
<trid="row_4_"class="even"><tdclass="entry"><spanstyle="width:16px;display:inline-block;"> </span><aclass="el"href="md__go_usage.html"target="_self">Use in Go</a></td><tdclass="desc"></td></tr>
<trid="row_5_"><tdclass="entry"><spanstyle="width:16px;display:inline-block;"> </span><aclass="el"href="md__java_usage.html"target="_self">Use in Java</a></td><tdclass="desc"></td></tr>
<trid="row_7_"><tdclass="entry"><spanstyle="width:16px;display:inline-block;"> </span><aclass="el"href="md__white_paper.html"target="_self">FlatBuffers white paper</a></td><tdclass="desc"></td></tr>
<trid="row_9_"><tdclass="entry"><spanstyle="width:16px;display:inline-block;"> </span><aclass="el"href="md__grammar.html"target="_self">Formal Grammar of the schema language</a></td><tdclass="desc"></td></tr>
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.