Commit Graph

939 Commits

Author SHA1 Message Date
Björn Harrtell
760c657551 [TS/JS] New gen TS code gen (#6302)
* TS/ES6 modules spike iteration 1

* Initial modularized dasherized output

* Remove obsoleted parts and namespace wrapping

* Use _flatbuffers_ prefix

* First part of imports logic

* Second part of imports logic

* Fix TS/JS code removal mixup

* Alias imported symbols if same name from different namespaces and some fixes

* Use star import for bare imports

* Fix messed up string concat

* var to const and remove not needed semi

* Remove some cases of ns prefixing

* Add missing space

* Cleanups

* Completed initial import tracking logic

* Compilable output

* Adjust TypeScriptTest and dependents to work

* Use local flatbuffers package for tests

* Refactor away use of any

* Remove obsolete imported_fileset and reexport_map

* Still need any and fix JavaScriptTest.sh

* Fix test runs out of the box

* Temp add generated files

* TypeScriptTest replaces JavaScriptTest and cleanups

* Also remove reference to JavaScriptTest in TestAll.sh

* Remove old generated ts/js files

* Remove use of --js in generate_code scripts

* idl_gen_js_ts to idl_gen_ts and removal of js gen

* Remove obsoleted options

* Fix obsolete ts test detection

* Tweak ts compilation be as strict as possible

* Remove jsdoc type annotation generation

* Generated test ts files

* Fix search and replace messup

* Regenerated ts test output

* Use CharToLower

* Use normal for loop

* Rework namespacedir

* Revert "Rework namespacedir"

This reverts commit 6f4eb0104ceeb86011bb076ebca901138c48e068.

* Revert "Use normal for loop"

This reverts commit 676b2135bfaa1853dfbb06c92b5c16a0d81bb13a.

* Revert "Use CharToLower"

This reverts commit 2d08648d0d72d0af201fad80d54cdc76412b35e9.

* Again do rework but correct

* Avoid runtime cast

* Fix test runs

* Also add npm install to get tsc

* Bump node test versions

* for range to std for loop

* Clang format

* Missed one clang format

* Move accessor to later

* Attempt to make windows version of TypeScriptTest

* Want to see the output

* Try to get newer node at appveyor

* Style changes
2021-01-19 12:51:13 -08:00
Vladimir Glavnyy
75c859e98f [idl_parser] Improve symbols lookup thru parent namespaces (#6407)
Added the new method LookupTableByName for searching symbols in parent namespaces.
This method speedup (x50 or greater) symbol resolution (enum or struct) in parent namespaces.
The speedup was measured without `table.empty()` guard condition.

This method should suppress fuzzer timeout issue without artificial limits for nested namespaces (https://oss-fuzz.com/testcase-detail/6244168439169024).

Additionally, this commit speedup (x5) the GetFullyQualifiedName method by removing unnecessary temporary std::string object.
2021-01-19 12:49:24 -08:00
Mark Spatz
91b0958c43 Search for includes in the directory containg the current file (#6371)
* [codegen] Search for includes in the directory containg the current file

This matches the behavior of a C preprocessor, see:
https://gcc.gnu.org/onlinedocs/cpp/Search-Path.html

* Skip FileExists() when file is guaranteed to not exist

* end comment with period to match others
2021-01-19 12:26:34 -08:00
Björn Harrtell
c81cf82492 [TS/JS] New gen TS code gen prequel to preserve history (#6404)
* Rename idl_gen_js_ts to idl_gen_ts to preserve history

* Change build related files
2021-01-14 11:34:44 -08:00
Casper
8573108bbe Unset FieldDef.optional if its key (#6402)
* Unset FieldDef.optional if its key

* removed StringKey table, just removed an extra "required, key"

* removed extra newline

Co-authored-by: Casper Neo <cneo@google.com>
2021-01-12 14:06:43 -05:00
Casper
408cf58024 Fix Rust UB problems (#6393)
* Fix miri problems by assuming alignment is 1 in rust

* Removed is_aligned fn from rust verifier.

* Add back is_aligned, but make it w.r.t. buffer[0]

* touch unused variable

* touch unused variable

* +nightly

* Move Rust miri testing into its own docker

* fix bash

* missing one endian conversion

* fix endianness2

* format stuff

Co-authored-by: Casper Neo <cneo@google.com>
2021-01-11 15:24:52 -05:00
Dan Field
809fe49c7a Fix up scripts, fix generated enum value for strong mode, regenerate files (#6389)
* Fix up scripts, fix generated enum value for strong mode, regenerate files

* missing files

* GH action

* remove action
2021-01-07 14:51:52 -08:00
Vladimir Glavnyy
83ce29cc22 [C++, JSON] Fix nullptr access when reading a key with a default value. (#6375)
This commit fixes handling of default and NULL `key` fields in `Parser::ParseVector` (#5928).

The JSON generator updated. It outputs `key` fields even if the `--force-defaults` option is inactive.

Additional test cases for `key` added.
2021-01-07 11:24:59 -08:00
Richard A Hofer
4363c1d2cb Fix generated EndVector. (#6385) 2021-01-05 14:45:23 -08:00
Stefan de Konink
1bf1ec0270 Implements type promotion for Java enum (#6382)
* Implements type promotion for Java enum as suggested in #3890, #5111, #6369

* After generate_code.sh
2021-01-05 11:11:29 -08:00
Kamil Rojewski
46545e6273 fixed warnings (#6355)
* semver-compatible deprecations

* removed unneeded lifetimes (+ clippy warnings)

* silenced too many args warning
2021-01-04 14:11:59 -08:00
Vladimir Glavnyy
82836a62be [idl_parser] Improve stack overflow protection (#6364)
* [idl_parser] Improve stack overflow protection

Add stack overflow protection for Flexbuffer and nested Flatbuffer parsers.
Replaces the `Recurse()` method by the new ParseDepthGuard RAII class.

* Remove move operator from Parser.

It was wrong decision to add move ctor and assignment into Parser class.
These operators will make it extremely difficult to add constant or reference fields in the future.

* Remove ';' from definition of FLATBUFFERS_DELETE_FUNC

* Format code

* Make this PR compatible with MSVC2010 (it doesn't support inherited ctor)
2021-01-04 12:39:12 -08:00
Vladimir Glavnyy
e7430bbebd [idl_parser] Check the range of explicitly set field's id value (#6363)
* [idl_parser] Check the range of explicitly set field's id value

The explicitly set `id` attribute should be a non-negative value of the `voffset_t` type.

* Format FieldIdentifierTest()
2021-01-04 12:33:31 -08:00
Richard A Hofer
24dd85fd28 Generate code to encode and decode nested flatbuffers in Python. (#6354)
* Generate code to encode and decode nested flatbuffers in Python.

* Delete accidental trailing whitespace.

* Fully delete trailing whitespace.
2021-01-04 12:18:19 -08:00
Kamil Rojewski
be37d4da14 include_prefix support for rust (#6330)
* include_prefix support for rust

* include_prefix support for java

* formatting fixes

* Revert "include_prefix support for java"

* style fix
2020-12-30 12:02:36 -08:00
mustiikhalil
4e79d129cb [Swift] Rebuild the way swift handles structs from scratch (#6326)
* Rebuild the way swift handles structs from scratch

* Updates docs, and sample binary

* Replaces InMemory to Mutable

* Migrates docs from inmemory

* use inline for some functions

* Renamed Mutable objects

* Updates documentation
2020-12-17 14:55:32 -08:00
mqy
05192553f4 Fix typos in usage/comments; Make rust generator respect to --filenam… (#6342)
* Fix typos in usage/comments; Make rust generator respect to --filename-suffix

* run clang format
2020-12-17 13:15:27 -08:00
Richard A Hofer
f8b203c9c4 Add vectorNumElements attribute to Builder for simpler vector creation. (#6328)
* Add vectorNumElements attribute to Builder for simpler vector creation.

This adds a default to EndVector which should simplify its use.

* Update tutorial to reflect new default arg in Python EndVector.

* Remove optional argument to Python EndVector.

* Add generated files.

* Unset Builder.vectorNumElems when not in use.
2020-12-14 12:00:28 -08:00
Stefan F
65c415911a Generate nullable properties in C# object-based API for optional scalars. (without -gen-mutable) (#6273)
* Added missing EndTable() call to VerifyObject()

VerifyObject called VerifyTableStart() but not EndTable(). This made Verifier::VerifyComplexity() increase depth_ with each table, not with the depth of tables.

https://groups.google.com/forum/#!topic/flatbuffers/OpxtW5UFAdg

* Added Check to VerifyAlignment

https://stackoverflow.com/questions/59376308/flatbuffers-verifier-returns-false-without-any-assertion-flatbuffers-debug-veri

* Add GetStringView (Convenience function to get string_view from a String returning an empty string_view on null pointer) like GetString, GetCstring

* flatc should warn, when an attribute is attached more than once.

flatc.exe -b duplicate.fbs
warning: duplicate.fbs(5, 36): warning: attribute already found: priority

duplicate.fbs:

namespace MyGame;

attribute "priority";

table Monster (priority:1, priority:2) {
}

root_type Monster;

* flatc should support --binary --schema with optional scalar fields.

This fixes 'error: Optional scalars are not yet supported in at least one the of the specified programming languages.' when calling flatc.exe --binary --schema with a schema containing optional scalars.

* Generate nullable properties in C# object-based API for optional scalars.
tests\generate_code.bat extended to test this.

Ran tests\generate_code.bat
Ran tests\Flatbuffers.Test\NetTest.bat

* %TEST_BASE_FLAGS% replaced with --gen-object-api in generate_code.bat, because only this is part of this PR. Added this same flag to generate_code.sh

* generate_code.bat and generate_code.sh changed to only test c# with object based api.
2020-12-10 16:50:30 -08:00
tira-misu
9fca5e4f42 Add flatc option to inhibit all warnings #6005 (#6301)
* Fix C/C++ Create<Type>Direct with sorted vectors

If a struct has a key the vector has to be sorted. To sort the vector
you can't use "const".

* Changes due to code review

* Improve code readability

* Add generate of JSON schema to string to lib

* option indent_step is supported

* Remove unused variables

* Fix break in test

* Fix style to be consistent with rest of the code

* Add option --no-warnings to inhibit all warnings

* Fix order of member initialization

* Add documentation for --no-warnings
2020-12-10 14:03:53 -08:00
Casper
442949bc11 Rust Flatbuffers Verifier (#6269)
* Updated comments and fixed a fundemental type error.

* bump rust flatbuffers semver

* Initial commit with verifier, need to clean up

* Verifier tested. Needs clean up and refactoring.

* Display for InvalidFlatbuffer and better errors for strings

* SimpleToVerify, some refactoring

* Combined VerifierType TableAccessorFuncBody into FollowType

* scrub todos

* Update Rust get_root functions.

There are 6 variants, with verifier options, default verifier options
and no verification "fast".

* Rename root fns

* inline

* Update to use thiserror

* fix for bad compiler

* improve error formatting

* Replace multiply with saturating_multiply

* saturating adds too

* Add docs disclaiming experimental verification system

Co-authored-by: Casper Neo <cneo@google.com>
2020-12-07 18:37:51 -05:00
Richard A Hofer
fc960f3670 Add default to offset param of Python generated GetRootAs (#6312) 2020-12-07 11:34:07 -08:00
Vladimir Glavnyy
7f33cf682a [C++] Switch flatc to --cpp-std c++11 C++ code generator (#6306)
Prior to this commit the default C++ code generator was `c++0x`.
A code generated with `c++0x` code-gen might have a vulnerability (undefined behavior) connected evolution of enums in a schema. This UB could break the backward compatibility if previously generated code casts an unknown enumerator to enum type that knows nothing about future enumerators added to the schema.

The main differences between `c++0x` and `c++11`:

- generated enums use explicitly declared underlying type;
- generated object-API tables don't declare default ctor() explicitly, instead of it default data member initializers are generated.

Please use `flatc --cpp-std c++0x` option for backward compatibility with old compilers.
2020-12-07 11:19:36 -08:00
Vladimir Glavnyy
8d9eae9ac9 [idl_parser] Unify parsing of NaN values read from .fbs and .json files (#6296)
This commit unifies parsing of NaN values read
from .fbs and .json files by converting them to unsigned NaN.
2020-12-07 11:12:58 -08:00
Vladimir Glavnyy
c27bc2d76f [C++] Add ParseJson(), Parser(Parser&&), update fuzzers (#6284)
- add a new method ParseJson to minimize failures during fuzzing
- add default (conditional) move-constructor for Parser
- add a new monster_fuzzer
- switch fuzzers to C++17 and `test/cpp17` generated code
2020-11-23 10:17:44 -08:00
Wouter van Oortmerssen
bc518a5127 Fixed FlexBufferBuilder asserting on duplicate keys 2020-11-23 09:02:40 -08:00
Austin Schuh
e58c182443 Add --require-explicit-ids to require explicit ids (#6277)
* Add --require-explicit-ids to require explicit ids

We just got bit by a well intentioned developer forgetting that field
order by default is the field index.  3 people missed it in review.

I'm looking at ways to make it harder to mess up.  We are requesting
that developers explicitly id all fields in tables.  Automatic (opt in
for others) enforcement of this will help the effort succeed.  This
patch adds a command line flag which lets the user require ids on all
fields in tables.

* Added docs to Compiler.md as well
2020-11-19 17:16:45 -08:00
schoetbi
69a8b2a579 idl_gen_json_schema.cpp: Changed generation of array element types (#6253)
* idl_gen_json_schema.cpp: Changed generation of array element types
#6175

* idl_gen_json_schema.cpp: Simplified indent generation as suggested by @vglavnyy
#6175
2020-11-19 16:17:03 -08:00
mustiikhalil
537212afee [Swift] Adds a format file and reformats the swift project (#6250)
* Formats files & adds licence

* Revert arrays

* Keeps array indentation as is

* Adds licence to code formatter

* Updates code generators
2020-11-16 19:52:38 +03:00
mustiikhalil
6764f25d91 Adds a fix for enum generation (#6263) 2020-11-15 17:50:12 +03:00
Jason Lenz
de1f0342c8 Remove _POSIX_C_SOURCE and _XOPEN_SOURCE definitions when compiling o… (#6205)
* Remove _POSIX_C_SOURCE and _XOPEN_SOURCE definitions when compiling on OpenBSD

* Only define _POSIX_C_SOURCE and _XOPEN_SOURCE for mingw/cygwin platforms

* Only define POSIX statements for mingw/cygwin/qnx platforms
2020-11-12 10:12:32 -08:00
Stefan F
d0d51e2a5c flatc should support --binary --schema with optional scalar fields. (#6252)
* Added missing EndTable() call to VerifyObject()

VerifyObject called VerifyTableStart() but not EndTable(). This made Verifier::VerifyComplexity() increase depth_ with each table, not with the depth of tables.

https://groups.google.com/forum/#!topic/flatbuffers/OpxtW5UFAdg

* Added Check to VerifyAlignment

https://stackoverflow.com/questions/59376308/flatbuffers-verifier-returns-false-without-any-assertion-flatbuffers-debug-veri

* Add GetStringView (Convenience function to get string_view from a String returning an empty string_view on null pointer) like GetString, GetCstring

* flatc should warn, when an attribute is attached more than once.

flatc.exe -b duplicate.fbs
warning: duplicate.fbs(5, 36): warning: attribute already found: priority

duplicate.fbs:

namespace MyGame;

attribute "priority";

table Monster (priority:1, priority:2) {
}

root_type Monster;

* flatc should support --binary --schema with optional scalar fields.

This fixes 'error: Optional scalars are not yet supported in at least one the of the specified programming languages.' when calling flatc.exe --binary --schema with a schema containing optional scalars.
2020-11-12 10:06:09 -08:00
Derek Bailey
febb9d87c9 Union As Accessors for C# (#6251)
* Union As Accessors for C#

* Changed loop to be compatible with older compilers

* errant change fix
2020-11-12 09:28:21 -08:00
Casper
46a8c7e958 Added required-nested-flatbuffer to monster_test and fixed rust (#6236)
Co-authored-by: Casper Neo <cneo@google.com>
2020-11-05 13:58:06 -08:00
Charlie Yin
c9b29d0885 Support size-prefixed buffers and add tests for size-prefixed messages (#6232) 2020-11-05 11:23:56 -08:00
Derek Bailey
ea92a668d8 [C#] Optional Scalars (#6217)
* [C#] Optional Scalars

* Moved scalar optional check to avoid null enum conversion
2020-10-30 13:18:59 -07:00
Casper
b08b0a4402 Implement Debug trait for Rust flatbuffers. (#6207)
* Refactor idl_gen_rust to a ForAllX continuation pattern.

* Removed unneeded SetValue and updated sample rust gencode

* Make Rust flatbuffers print right

* Generated code and removed unnecessary trait constraint

* bumped rust version. Release required

* removed an unwrap in Rust Debug-print unions

* Tested formatting flatbuffers in rust.

* Set float precision in flaky debug-print test

* impl Debug for structs too

Co-authored-by: Casper Neo <cneo@google.com>
2020-10-29 12:57:29 -07:00
Kamil Rojewski
a0182cdb11 optional scalars for ts/js (#6215)
* optional scalars for ts/js

* removed range based for

* removed range based for
2020-10-28 09:45:23 -07:00
Paulo Pinheiro
f9a18ea635 [Java] Implement optional scalars (#6212)
Java implementation of optional scalars, following issue #6014.
2020-10-26 17:46:50 -07:00
Derek Bailey
914c646014 Removed C# references from java generator. Move annotations closer to definitions (#6204) 2020-10-25 21:42:29 -07:00
Casper
e68e8d7de9 Refactor idl_gen_rust (#6206)
* Refactor idl_gen_rust to a ForAllX continuation pattern.
* Updated rust sample code

Co-authored-by: Casper Neo <cneo@google.com>
2020-10-24 15:46:55 -07:00
Casper
fe8e3c7e5d Mass Refactoring to use IsString and other BASE_TYPE helpers (#6193)
* Updated SupportsAdvancedUnionFeatures to look out for string

* Mass refactoring to use BASE_TYPE helper functions.

Co-authored-by: Casper Neo <cneo@google.com>
2020-10-20 08:00:31 -07:00
Casper
8f6fa4b715 Updated SupportsAdvancedUnionFeatures to look out for string (#6190)
Co-authored-by: Casper Neo <cneo@google.com>
2020-10-19 13:16:00 -07:00
Casper
9fa1d27059 Rework enums in rust. (#6098)
* Rework enums in rust.

They're now a unit struct, rather than an enum. This is a
backwards incompatible change but the previous version had UB
and was also backwards incompatible so...

* Update and test sample rust flatbuffers

* Use bitflags crate to properly support rust enums.

Previously, the bitflags attribute was just ignored. This is a breaking change
as the bitflgs API is not like a normal rust enum (duh).

* variant_name() -> Option<_>

* repr transparent

* Reexport bitflags from flatbuffers

* Make bitflags constants CamelCase, matching normal enums

* Deprecate c-style associated enum constants

Co-authored-by: Casper Neo <cneo@google.com>
2020-10-19 11:40:03 -07:00
schoetbi
a402b3abae idl_gen_json_schema Fix generation of arrays of enums (#6184)
* idl_gen_json_schema.cpp: Fixed generation of arrays of enums
#6175

* Fixed failing unit tests

* GenBaseType generate "integer".
Fixes #6066

* Ran tests/generate_code.

* Removed modern R"()" strings

* changed std::to_string to NumToString
2020-10-16 10:43:09 -07:00
tira-misu
5cd7137103 Add generation of JSON Schema to library (#6165)
* Fix C/C++ Create<Type>Direct with sorted vectors

If a struct has a key the vector has to be sorted. To sort the vector
you can't use "const".

* Changes due to code review

* Improve code readability

* Add generate of JSON schema to string to lib

* option indent_step is supported

* Remove unused variables

* Fix break in test

* Fix style to be consistent with rest of the code
2020-10-15 15:28:25 -07:00
astange
4ec5e8db90 [C++] Add option to not generate direct copy methods. (#6166)
* Add option to not generate direct copy methods.

The direct copy methods generated by flatc utilize std::vector which isn't allowed on some embedded systems. Permit users of the compiler to not generate these methods so they don't have to be stubbed out.

* Update docs for no-cpp-direct-copy option.
2020-10-12 12:25:10 -07:00
Vladimir Glavnyy
04bec23a37 Add Array initialization from struct constructor (#5865) (#6147)
- add flatbuffers::span
- add new constructor for `struct` with `array`
- add some test for flatbuffers::span and 'arrays_test.fbs'
2020-10-12 12:24:18 -07:00
Casper
543c1bbeba Fixed rust nested flatbuffers for tables other than self (#6062)
* Fixed rust nested flatbuffers for tables other than self

* replaced lifetimes

* Use WrapInNameSpace and also update samples

Co-authored-by: Casper Neo <cneo@google.com>
2020-10-10 11:49:02 -07:00
mustiikhalil
7b9e61fccf [TS] GRPC Implementation (#6141)
* GRPC implementation for Typescript

* Fixes a couple of issues

* Finished implementing the typescript support for grpc

* Updated generated code

* Fixes CI
2020-10-07 09:56:30 -07:00