Commit Graph

839 Commits

Author SHA1 Message Date
Stefan F
65c415911a Generate nullable properties in C# object-based API for optional scalars. (without -gen-mutable) (#6273)
* Added missing EndTable() call to VerifyObject()

VerifyObject called VerifyTableStart() but not EndTable(). This made Verifier::VerifyComplexity() increase depth_ with each table, not with the depth of tables.

https://groups.google.com/forum/#!topic/flatbuffers/OpxtW5UFAdg

* Added Check to VerifyAlignment

https://stackoverflow.com/questions/59376308/flatbuffers-verifier-returns-false-without-any-assertion-flatbuffers-debug-veri

* Add GetStringView (Convenience function to get string_view from a String returning an empty string_view on null pointer) like GetString, GetCstring

* flatc should warn, when an attribute is attached more than once.

flatc.exe -b duplicate.fbs
warning: duplicate.fbs(5, 36): warning: attribute already found: priority

duplicate.fbs:

namespace MyGame;

attribute "priority";

table Monster (priority:1, priority:2) {
}

root_type Monster;

* flatc should support --binary --schema with optional scalar fields.

This fixes 'error: Optional scalars are not yet supported in at least one the of the specified programming languages.' when calling flatc.exe --binary --schema with a schema containing optional scalars.

* Generate nullable properties in C# object-based API for optional scalars.
tests\generate_code.bat extended to test this.

Ran tests\generate_code.bat
Ran tests\Flatbuffers.Test\NetTest.bat

* %TEST_BASE_FLAGS% replaced with --gen-object-api in generate_code.bat, because only this is part of this PR. Added this same flag to generate_code.sh

* generate_code.bat and generate_code.sh changed to only test c# with object based api.
2020-12-10 16:50:30 -08:00
David P. Sicilia
a9e91116d2 [Python] Commit some orphan python genfile diffs. (#6325) 2020-12-10 16:43:40 -08:00
Vladimir Glavnyy
92a806b4e8 [fuzzer] Rename fuzzing dictionaries for oss-fuzz (#6318)
This commit makes the names of fuzzing dictionaries the same as the target binary names.
Also it explicitly limits size of test inputs to prevent failures in `regex` and fuzzing time-outs.
2020-12-10 14:02:01 -08:00
Casper
442949bc11 Rust Flatbuffers Verifier (#6269)
* Updated comments and fixed a fundemental type error.

* bump rust flatbuffers semver

* Initial commit with verifier, need to clean up

* Verifier tested. Needs clean up and refactoring.

* Display for InvalidFlatbuffer and better errors for strings

* SimpleToVerify, some refactoring

* Combined VerifierType TableAccessorFuncBody into FollowType

* scrub todos

* Update Rust get_root functions.

There are 6 variants, with verifier options, default verifier options
and no verification "fast".

* Rename root fns

* inline

* Update to use thiserror

* fix for bad compiler

* improve error formatting

* Replace multiply with saturating_multiply

* saturating adds too

* Add docs disclaiming experimental verification system

Co-authored-by: Casper Neo <cneo@google.com>
2020-12-07 18:37:51 -05:00
Vladimir Glavnyy
bc7eb8adeb [fuzzer] Fix mistakes in the parser and scalar fuzzers. (#6314)
The flatbuffers::Parser::Parse() isn't an idempotent method for schema parsing.
This commit removes a wrong for-loop that tried to check the same schema twice.
2020-12-07 11:47:33 -08:00
Vladimir Glavnyy
f437f0f7ed [fuzzer] Fix loading of schema in monster_fuzzer (#6308)
This is fix for (https://oss-fuzz.com/testcase-detail/6251772204810240)
2020-12-07 11:21:23 -08:00
Vladimir Glavnyy
7f33cf682a [C++] Switch flatc to --cpp-std c++11 C++ code generator (#6306)
Prior to this commit the default C++ code generator was `c++0x`.
A code generated with `c++0x` code-gen might have a vulnerability (undefined behavior) connected evolution of enums in a schema. This UB could break the backward compatibility if previously generated code casts an unknown enumerator to enum type that knows nothing about future enumerators added to the schema.

The main differences between `c++0x` and `c++11`:

- generated enums use explicitly declared underlying type;
- generated object-API tables don't declare default ctor() explicitly, instead of it default data member initializers are generated.

Please use `flatc --cpp-std c++0x` option for backward compatibility with old compilers.
2020-12-07 11:19:36 -08:00
Vladimir Glavnyy
8d9eae9ac9 [idl_parser] Unify parsing of NaN values read from .fbs and .json files (#6296)
This commit unifies parsing of NaN values read
from .fbs and .json files by converting them to unsigned NaN.
2020-12-07 11:12:58 -08:00
Vladimir Glavnyy
c27bc2d76f [C++] Add ParseJson(), Parser(Parser&&), update fuzzers (#6284)
- add a new method ParseJson to minimize failures during fuzzing
- add default (conditional) move-constructor for Parser
- add a new monster_fuzzer
- switch fuzzers to C++17 and `test/cpp17` generated code
2020-11-23 10:17:44 -08:00
schoetbi
69a8b2a579 idl_gen_json_schema.cpp: Changed generation of array element types (#6253)
* idl_gen_json_schema.cpp: Changed generation of array element types
#6175

* idl_gen_json_schema.cpp: Simplified indent generation as suggested by @vglavnyy
#6175
2020-11-19 16:17:03 -08:00
mustiikhalil
537212afee [Swift] Adds a format file and reformats the swift project (#6250)
* Formats files & adds licence

* Revert arrays

* Keeps array indentation as is

* Adds licence to code formatter

* Updates code generators
2020-11-16 19:52:38 +03:00
AdamKorcz
d6a8dbd26f Experimental fix for failing oss-fuzz coverage build (#6259) 2020-11-13 15:48:05 -08:00
AdamKorcz
a49531414b Update to flags in fuzzing-cmake file (#6256) 2020-11-12 10:21:34 -08:00
Derek Bailey
febb9d87c9 Union As Accessors for C# (#6251)
* Union As Accessors for C#

* Changed loop to be compatible with older compilers

* errant change fix
2020-11-12 09:28:21 -08:00
Casper
46a8c7e958 Added required-nested-flatbuffer to monster_test and fixed rust (#6236)
Co-authored-by: Casper Neo <cneo@google.com>
2020-11-05 13:58:06 -08:00
Charlie Yin
c9b29d0885 Support size-prefixed buffers and add tests for size-prefixed messages (#6232) 2020-11-05 11:23:56 -08:00
Derek Bailey
ea92a668d8 [C#] Optional Scalars (#6217)
* [C#] Optional Scalars

* Moved scalar optional check to avoid null enum conversion
2020-10-30 13:18:59 -07:00
Casper
b08b0a4402 Implement Debug trait for Rust flatbuffers. (#6207)
* Refactor idl_gen_rust to a ForAllX continuation pattern.

* Removed unneeded SetValue and updated sample rust gencode

* Make Rust flatbuffers print right

* Generated code and removed unnecessary trait constraint

* bumped rust version. Release required

* removed an unwrap in Rust Debug-print unions

* Tested formatting flatbuffers in rust.

* Set float precision in flaky debug-print test

* impl Debug for structs too

Co-authored-by: Casper Neo <cneo@google.com>
2020-10-29 12:57:29 -07:00
Kamil Rojewski
a0182cdb11 optional scalars for ts/js (#6215)
* optional scalars for ts/js

* removed range based for

* removed range based for
2020-10-28 09:45:23 -07:00
Derek Bailey
0dfcc0a378 Adds NetTest.bat to run .NET Core tests on Windows (#6216) 2020-10-27 22:51:58 -07:00
Paulo Pinheiro
f9a18ea635 [Java] Implement optional scalars (#6212)
Java implementation of optional scalars, following issue #6014.
2020-10-26 17:46:50 -07:00
mustiikhalil
c7586e85aa Empties the sharedString map on reset on go and csharp (#6187)
Fixes go tests
2020-10-26 12:38:24 -07:00
Derek Bailey
914c646014 Removed C# references from java generator. Move annotations closer to definitions (#6204) 2020-10-25 21:42:29 -07:00
Paulo Pinheiro
1606fb6375 Kotlin test optional enum (#6201)
* Add test for optional enums in Kotlin

* Rename optional_scalars2.fbs into optional_scalars.fbs

Also updated all references in the project to point to
"optional_scalars.fbs" instead of "optional_scalars2.fbs".
2020-10-22 13:36:02 -07:00
Casper
9fa1d27059 Rework enums in rust. (#6098)
* Rework enums in rust.

They're now a unit struct, rather than an enum. This is a
backwards incompatible change but the previous version had UB
and was also backwards incompatible so...

* Update and test sample rust flatbuffers

* Use bitflags crate to properly support rust enums.

Previously, the bitflags attribute was just ignored. This is a breaking change
as the bitflgs API is not like a normal rust enum (duh).

* variant_name() -> Option<_>

* repr transparent

* Reexport bitflags from flatbuffers

* Make bitflags constants CamelCase, matching normal enums

* Deprecate c-style associated enum constants

Co-authored-by: Casper Neo <cneo@google.com>
2020-10-19 11:40:03 -07:00
schoetbi
a402b3abae idl_gen_json_schema Fix generation of arrays of enums (#6184)
* idl_gen_json_schema.cpp: Fixed generation of arrays of enums
#6175

* Fixed failing unit tests

* GenBaseType generate "integer".
Fixes #6066

* Ran tests/generate_code.

* Removed modern R"()" strings

* changed std::to_string to NumToString
2020-10-16 10:43:09 -07:00
Vladimir Glavnyy
04bec23a37 Add Array initialization from struct constructor (#5865) (#6147)
- add flatbuffers::span
- add new constructor for `struct` with `array`
- add some test for flatbuffers::span and 'arrays_test.fbs'
2020-10-12 12:24:18 -07:00
Casper
543c1bbeba Fixed rust nested flatbuffers for tables other than self (#6062)
* Fixed rust nested flatbuffers for tables other than self

* replaced lifetimes

* Use WrapInNameSpace and also update samples

Co-authored-by: Casper Neo <cneo@google.com>
2020-10-10 11:49:02 -07:00
mustiikhalil
7b9e61fccf [TS] GRPC Implementation (#6141)
* GRPC implementation for Typescript

* Fixes a couple of issues

* Finished implementing the typescript support for grpc

* Updated generated code

* Fixes CI
2020-10-07 09:56:30 -07:00
Casper
3359e3042f Moved C++ to optional_scalars2 and added some tests. (#6162)
* Moved C++ to optional_scalars2 and added some tests.

Also deleted unused optional_scalars_generated.lobster

* Fixed whitespece in C++ gencode & fixed BUILD file

* Moved C++ onto optional_scalars2 in the .bat file

Co-authored-by: Casper Neo <cneo@google.com>
2020-10-07 09:46:09 -07:00
mrmarkwell
08943aa26f Flatbuffer C++ UnpackTo optimization for vectors of non-bool bytes. (#6154)
UnpackTo copies vector elements one-by-one, which can be very inefficient depending on the quality of the compiler optimizations performed. This change updates the operation for vectors of bytes that aren't enums to use 'std::copy', which is usually highly optimized.

vectors of types that are more than one byte can't be optimized in this way because of the endianness of the serialized bytes vs. the target architecture endianness.

vectors of enums can't be optimized because they are required to be static_cast into the appropriate enum type when stored in the vector.

vectors of bools can be optimized in most cases, but since the standard
allows std::vector<bool> template specialization for space-savings,
std::copy doesn't work on every implementation (looking at you
Microsoft). Thus, this optimization is skipped for vector<bool>.

For a specific example, this improves the latency of unpacking large buffers on the Hexagon DSP by about 10x.

Co-authored-by: Matthew Markwell <markwell@google.com>
2020-10-06 10:56:45 -07:00
mustiikhalil
5975658ebd Enables optional enums in swift (#6160)
[Swift] Enables optional enums
2020-10-06 19:00:14 +03:00
Wouter van Oortmerssen
5d3cf440e5 Updated Lobster test for optional bools/enums
The codegen for this was already correct, just added more tests
2020-10-02 10:53:47 -07:00
Casper
8ec8322f09 Ruopt enum (#6156)
* Rust support for optional enums

* make optional_scalars2 for languages that support optional enums

Co-authored-by: Casper Neo <cneo@google.com>
2020-10-02 10:30:39 -07:00
Vladimir Glavnyy
bbcc85fd4c Fix generation of C++ code with Optional<Enum> (#6155) 2020-10-02 10:27:30 -07:00
Anass Al
689bfafa7e [Python/JS/TS] Codegen SizeOf method for structs (#6136)
* [Python] Codegen SizeOf classmethod for structs

This codegens a `SizeOf()` classmethod for all structs since we can't
determine the size of a FlatBuffer generated struct from Python otherwise.

* [JS/TS] Codegen sizeOf static method for structs

This codegens a `sizeOf()` static method for all structs since we can't
determine the size of a FlatBuffer generated struct from JavaScript or
TypeScript otherwise.
2020-09-28 08:54:50 -07:00
石晓伟
641309a5bf unix2dos on tests/FlatBuffers.Test/FlatBuffers.Core.Test.csproj (#6133) 2020-09-28 08:10:06 -07:00
春条
e0bbaa6f9c [C#]Change to ENABLE_SPAN_T that doesn't require UNSAFE_BYTEBUFFER. (#6073)
* ENABLE_SPAN_T doesn't require UNSAFE_BYTEBUFFER.

Change to ENABLE_SPAN_T that doesn't require UNSAFE_BYTEBUFFER.

* Selectable framework.

Changed target framework to allow selection of 2.0 or 2.1 (or higher)

* Added target framework version check.

* Add core test project.

* Added run on .Net Core.
2020-09-23 13:16:00 -07:00
Mark Nauwelaerts
34d67b425e Minireflect fixed array (#6129)
* CMakeLists: also really generate optional_scalars as needed by test elsewhere

* [C++] Handle fixed-length array in minireflection

Fixes #6128
2020-09-22 17:57:01 -07:00
Vladimir Glavnyy
eb686a86f3 Add missed file generated by optional_scalar.fbs (#6125) 2020-09-21 09:37:01 -07:00
Vladimir Glavnyy
8c67b5b129 Add support of Optional<T> scalars to C++ code generator (#6092) 2020-09-18 12:51:17 -07:00
Paulo Pinheiro
6228b66d3d [Kotlin] Support for optional scalars. (#6115)
More information on #6014
2020-09-17 23:31:51 -07:00
mustiikhalil
e1be8aaadd Bump version for latest swift version (#6121) 2020-09-17 12:28:42 -07:00
Björn Harrtell
94873e595c [JS/TS] Modernize TypeScript / JavaScript flatbuffers support (#6095) 2020-09-17 12:21:14 -07:00
Anass Al
b8e87fafe4 [JS] Add getFullyQualifiedName() (#6119)
This optionally generates a static `getFullyQualifiedName()` function to get
the fully-qualified name of a type in JavaScript and TypeScript in a similar
fashion to the C++ codegen.
2020-09-17 12:19:07 -07:00
mustiikhalil
89435303b7 [Swift] Migrates struct write APIS to write directly to the buffer (#6093)
* Migrates struct write APIS to in place APIS

* Fixes indentation in grpc swift
2020-09-17 17:10:59 +03:00
Casper
c75ae24293 Optional-ness in reflection (#6097)
* Optional scalars in reflection

* fixed name collision

* Remove code duplicated by merge

Co-authored-by: Casper Neo <cneo@google.com>
2020-09-11 14:20:32 -07:00
Casper
338944d3d9 Rename Nullable scalars to Optional scalars (#6112)
Co-authored-by: Casper Neo <cneo@google.com>
2020-09-10 13:04:36 -07:00
Maxim Zaks
92a8c1a0f2 [JS] FlexBuffers Fix for wrong type of offset and length values (#6107)
* Adding FlexBuffers support for Dart language

* Introduce snapshot method.

* Fix docu

* Replacing extension methods with static methods in order to support older Dart version

* Improving code based on PR feedback. Mainly rename refactoring.

* Addressing all PR feedback which does not need clarification

* exchange dynamic type with Object

* Adds better API documentation.
[] operator throws a very descriptive exception in case of a bad key.

* Implementation of JavaScript FlexBuffers decoder

* implements JS FlexBuffers builder

* replacing _toF32 with Math.fround

* Introducing test for BigInt number

* Moving functions from BitWitdth & ValueType object into BitWidthUtil and ValueTypeUtil accordingly.
Removing defensive checks.
Introducing fix for large int numbers by converting them to BigInt type.
Introducing handling for BigInt type in `add` method.
Using TextEncoder and Decoder to handle string / utf8 conversion.

* rename variable

* Lets user turn deduplication strategies for strings, keys and vector of keys off while building FlexBuffer.
Implements quick sort and choses quick sort if the number of keys is bigger then 20.
Removes unnecessary dict lookups in BitWidthUtil helper functions

* make iwidth and uwidth computation simpler and faster

* Making redInt and readUint a bit faster and shim the BigInt / BigUint usage

* Fixing a bug in FlexBuffers JS, where offsets and lengths are stored and read as int and not as uint values.

* Fixing a bug in FlexBuffers Dart, where offset and length values are stored and read as int values instead of uint values
2020-09-10 12:36:37 -07:00
Maxim Zaks
71aca81ff9 [JS] FlexBuffers Support (#5973)
* Adding FlexBuffers support for Dart language

* Introduce snapshot method.

* Fix docu

* Replacing extension methods with static methods in order to support older Dart version

* Improving code based on PR feedback. Mainly rename refactoring.

* Addressing all PR feedback which does not need clarification

* exchange dynamic type with Object

* Adds better API documentation.
[] operator throws a very descriptive exception in case of a bad key.

* Implementation of JavaScript FlexBuffers decoder

* implements JS FlexBuffers builder

* replacing _toF32 with Math.fround

* Introducing test for BigInt number

* Moving functions from BitWitdth & ValueType object into BitWidthUtil and ValueTypeUtil accordingly.
Removing defensive checks.
Introducing fix for large int numbers by converting them to BigInt type.
Introducing handling for BigInt type in `add` method.
Using TextEncoder and Decoder to handle string / utf8 conversion.

* rename variable

* Lets user turn deduplication strategies for strings, keys and vector of keys off while building FlexBuffer.
Implements quick sort and choses quick sort if the number of keys is bigger then 20.
Removes unnecessary dict lookups in BitWidthUtil helper functions

* make iwidth and uwidth computation simpler and faster

* Making redInt and readUint a bit faster and shim the BigInt / BigUint usage
2020-08-31 12:16:55 -07:00