The BytesConsumed function uses the `cursor_` to determine how many
bytes have been consumed by the parser, in case the user of the Parser
object wants to step over the parsed flatbuffer that is embedded in some
larger string. However, the `cursor_` is always one token ahead, so that
it can determine how to consume it. It points at the token that is about
to be consumed, which is ahead of the last byte consumed.
For example, if you had a string containing these two json objects and
parsed them...
"{\"key\":\"value\"},{\"key\":\"value\"}"
...then the `cursor_` would be pointing at the comma between the two
tables. If you were to hold a pointer to the beginning of the string and
add `BytesConsumed()` to it like so:
const char* json = // ...
parser.ParseJson(json);
json += parser.BytesConsumed();
then the pointer would skip over the comma, which is not the expected
behavior. It should only consume the table itself.
The solution is simple: Just hold onto a previous cursor location and
use that for the `BytesConsumed()` call. The previous cursor location
just needs to be set to the cursor_ location each time the cursor_ is
about to be updated. This will result in `BytesConsumed()` returning
the correct number of bytes without the off-by-one-token error.
Co-authored-by: Derek Bailey <derekbailey@google.com>
* add support for using array of scalar as key field
* update cmakelist and test.cpp to include the tests
* update bazel rule
* address comments
* clang format
* delete comment
* delete comment
* address the rest of the commnets
* address comments
* update naming in test file
* format build file
* buildifier
* make keycomparelessthan call keycomparewithvalue
* update to use flatbuffer array instead of raw pointer
* clang
* format
* revert format
* revert format
* update
* run generate_code.py
* run code generator
* revert changes by generate_code.py
* fist run make flatc and then run generate_code.py
Co-authored-by: Wen Sun <sunwen@google.com>
* Fix C/C++ Create<Type>Direct with sorted vectors
If a struct has a key the vector has to be sorted. To sort the vector
you can't use "const".
* Changes due to code review
* Improve code readability
* Add generate of JSON schema to string to lib
* option indent_step is supported
* Remove unused variables
* Fix break in test
* Fix style to be consistent with rest of the code
* [TS] Fix reserved words as arguments (#6955)
* [TS] Fix generation of reserved words in object api (#7106)
* [TS] Fix generation of object api
* [TS] Fix MakeCamel -> ConvertCase
* [C#] Fix collision of field name and type name
* [TS] Add test for struct of struct of struct
* Update generated files
* Add missing files
* [TS] Fix query of null/undefined fields in object api
* Put documentation to bfbs if it is not empty
* Fix monster test bfbs reference files
* Fix generated monster test files
Why they are different when generating it with linux and windows executable?
* Add support for proto 3 map to fbs gen
* Run clang-format
* Update proto golden test
* Rename variables
* Remove iostream
* Remove iostream
* Run clang format
Co-authored-by: Derek Bailey <derekbailey@google.com>
Previously when parsing a JSON representation of a Flatbuffer, the
parser required that the input string contain one and only one root
table. This change adds a flag that removes that requirement, so that
if a Flatbuffer table is embedded in some larger string the parser will
simply stop parsing once it reaches the end of the root table, and does
not validate that it has reached the end of the string.
This change also adds a BytesConsumed function, which returns the number
of bytes the parser consumed. This is useful if the table embedded in
some larger string that is being parsed, and that outer parser needs to
know how many bytes the table was so that it can step over it.
* [TS] Add support for fixed length arrays on Typescript (#5864) (#7021)
* Typescript / Javascript don't have fixed arrays but it is important to support these languages for compatibility.
* Generated TS code checks the length of the given array and do truncating / padding to conform to the schema.
* Supports the both standard API and Object Based API.
* Added a test.
Co-authored-by: Mehmet Baker <mehmet.baker@zerodensity.tv>
Signed-off-by: Bulent Vural <bulent.vural@zerodensity.tv>
Signed-off-by: Bülent Vural <bulent.vural@zerodensity.tv>
* Formatting & readability fixes on idl_gen_ts.cpp
Signed-off-by: Bülent Vural <bulent.vural@zerodensity.tv>
* Added array_test_complex.bfbs
Signed-off-by: Bülent Vural <bulent.vural@zerodensity.tv>
* TS arrays_test_complex: Remove bfbs and use fbs directly
Signed-off-by: Bülent Vural <bulent.vural@zerodensity.tv>
Signed-off-by: Bülent Vural <bulent.vural@zerodensity.tv>
* Bfbs Nim Generator
* Remove commented out tests
* add missing line to idl.h
* Commit python reflection changes
* Commit python reflection changes and move tests
* Remove default string addition
* Move tests to python file
* Fix element size check when element is table
* remove whitespace changes
* add element_type docs and commit further to namer and remove kkeep
* Bfbs Nim Generator
* Remove commented out tests
* add missing line to idl.h
* Commit python reflection changes
* Commit python reflection changes and move tests
* Remove default string addition
* Move tests to python file
* Fix element size check when element is table
* remove whitespace changes
* add element_type docs and commit further to namer and remove kkeep
* remove unused variables
* added tests to ci
* added tests to ci
* fixes
* Added reflection type Field, Variable to namer
* Moved reflection namer impl to bfbsnamer
* Remove whitespace at end of line
* Added nim to generated code
* Revert whitespace removal
Co-authored-by: Derek Bailey <derekbailey@google.com>
* Add support for metadata attributes for enum values (#7567)
* Fix path lookup in flatc test
* Try a fix for Windows paths
* Convert path to string to fix Windows error
* [C++] Add a failing unit test for #7516 (Rare bad buffer content alignment if sizeof(T) != alignof(T))
* [C++] Fix final buffer alignment when using an array of structs
* A struct can have an arbitrary size and therefore sizeof(struct) == alignof(struct)
does not hold anymore as for value primitives.
* This patch fixes this by introducing alignment parameters to various
CreateVector*/StartVector calls.
* Closes#7516
Added (for compiler versions that support it):
-Wmissing-declarations
-Wzero-as-null-pointer-constant
Then, fixes to problems identified by the extra warnings
Tested only on GCC 9.4.0
Adjusted the CPP code generator to output nullptr where appropriate,
to satisfy -Wzero-as-null-pointer-constant
Added a lot of 'static' declarations in front of functions,
to satisfy -Wmissing-declarations,
and wrap static function defs in anonymous namespaces.
There are advantages to both anonymous namespaces and static,
it seems that marking a function as static will not publish the name in
the symbol table at all, thus giving the linker less work to do.
* Implement optional scalars for JSON
* Add optional scalars JSON test
* Extend JSON optional scalars test to test without defaults
* Fix optional scalars in JSON for binary schema
Co-authored-by: Caleb Zulawski <caleb.zulawski@caci.com>
* Implement optional scalars for Python
* Use == for integer comparison, remove empty line
* Fix optional type hint
Co-authored-by: Caleb Zulawski <caleb.zulawski@caci.com>
* Unified name case conversion to single method
* Convert bfbs_gen to use ConvertCase
* convert rust to use ConvertCase
* Convert idl_parser to use ConvertCase
* Convert MakeScreamingCamel to ConvertCase
* Replaced MakeCamel with ConvertCase
* minor fixes
* [go] always write required types
* support optional scalars in go
* generate optional_scalars and monster_test
* restore original behavior for non-optional scalars
* add tests
* Add --warnings-as-errors to flatc compiler.
With this option set, flatc will return an error on parsing if
any warnings occurred.
* Add unit test for opts.warnings_as_errors.
* Change explicit option setting to default.
Parsing as bytes produces buffers that are unsafe to access unless passed thru a verifier,
whereas users could reasonably assume that any JSON parsed without errors is safe to access.
Users that still have legacy JSON files with such bytes in it will get a helpful error point them
to the option to turn on to have it work again.
* Make idl_parser deterministic
Some golden tests even exercise this [logic](df2df21ec1/tests/prototest/test.golden (L8)). Let's make the parser fully stable not depending on the implementation of std::sort
* Retry the ci
* initial hack to get new Lua generator into flatc
* Starting to output enum defs for Lua
* Continue to work on table generation for Lua
* Finished basic getter access for Lua
* Added ability to get object by index
* Finished struct builder
* aliased reflection to r
* finish table builder generation
* register requiring files
* better generated header info
* Tying up loose ends
* Updated reflection to handle struct padding
* Addd type sizes to reflection
* Fixed some vector indirect issues
* Lua tests passed
* Misc cleanup
* ci fixes 1
* ci fixes 2
* renaming
* up size of type sizes
* manually ran clang-format-11 -i src/idl_parser.cpp
* fixed some windows casting
* remove stupid auto import
* more static_casting
* remove std
* update other build environments
* remove scoped enums
* replaced std::to_string with NumToString
* more win fixes
* more win fixes
* replaced old lua with new
* removed auto import
* review responses
* more style fixes
* refactor bfbs_gen_len to use code +=
* added consts
* fix lambda capture for windows
* remove unused return type
* split flatbuffers.h into separate files
* wrong variable in cmakelists for android
* readded two accidentally deleted includes
* created buffer.h and moved buffer related things over
It seems like `--conform` already works for vectors of unions - there is
just a spurious check that prevents it from running. Fixes#6882
Also, if schemas do not conform, `flatc` no longer prints out the usage
(since the error is not due to bad usage). Fixes#6496
* Replace filenames in reflection with filenames+includes.
This is needed for some use cases and may be just useful metadata.
* deser files_included_per_file_
* check project_root
* fix bazel
* git clang format
Co-authored-by: Casper Neo <cneo@google.com>
* Add the file a symbol is declared in to Reflection
If we move a code-generator to depend on Reflection,
it may need to know which file something was declared in
to properly name generated files.
* Doc comments in reflection, and more precise tests
* Add --project-root flag to flatc, normalize declaraion_file to this root
* fix --project-root stuff
* posixpath
* fix scripts
* format
* rename --project-root to --bfbs-filenames
Also, make it optional, rather than defaulting to `./`, if its not
specified, then don't serialize the filenames.
* bfbs generation
* fix some tests
* uncomment a thing
* add to project root directory conditionally
* fix
* git clang format
* Added help description and removed != nullptr
* "
* Remove accidental change to docs
* Remove accidental change to docs
* Pool strings
Co-authored-by: Casper Neo <cneo@google.com>
* [idl_parser] Check structs and enums do not clash in a namespace
Uses fully qualified names to check for clashes within a given namespace whether explicitly defined or in the global namespace.
* [idl_parser] Move type name clash check to ParseEnum and ParseDecl
Change point at which parsing error is returned to ensure error is caught more generally. This change means that the error is returned after parsing the entirety of the offending duplicate rather than at the start when parsing it's name.
* [idl_parser] Add single and multi file type name clash tests
Adds a selection of tests for valid single file schemas with types that have the same name but are in different namespaces.
Adds a test for an a valid schema that spans two files with two types that have the same name but are in different namespaces.
Adds a test for an an invalid schema that spans two files with two types that have the same name and are in the same namespace.
* Add advance feature indicators to reflection
* deserialize too
* model advanced features as bitflags
* use uint64_t instead of AdvancedFeatures
* git clang format
* initialize advanced_features_
* remove whitespace
Co-authored-by: Casper Neo <cneo@google.com>
* Add support for fixed size arrays
* clang-format
* Update rust image to 1.51 to support const generics
* Handle correctly big endian
* Add fuzz tests and clean code
* Add struct fuzz test and optimize struct arrays for api
* Bump flatbuffers crate version
* disable clippy
* Vector of enum default
* swift and tests
* git clang format
* Rewrite enum parser checks
* Remove Voids from more_defaults
* vector enum swift
* remove vector accessor from swift
* clang format
Co-authored-by: Casper Neo <cneo@google.com>
* [idl_parser] Add kTokenNumericConstant token
This commit adds the new token for correct parsing of signed numeric constants.
Before this expressions `-nan` or `-inf` were treated as kTokenStringConstant.
This was ambiguous if a real string field parsed.
For example, `{ "text_field" : -name }` was accepted by the parser as valid JSON object.
Related oss-fuzz issue: 6200301176619008
* Add additional positive tests fo 'inf' and 'nan' as identifiers
* Rebase to HEAD
* Move processing of signed constants to ParseSingleValue method.
* Add missed `--cpp-static-reflection` (#6324) to pass CI
* Remove `flatbuffers.pc` from repository to unblock CI (#6455).
Probably the generated flatbuffers.pc should not be a part of repo.
* Fix FieldIdentifierTest()
This commit disable JSON parsing for an incomplete scheme if JSON object is embedded into one file with the scheme.
This should improve the quality of OSS-Fuzz inputs for the parser_fuzzer target.
