From 09a2999c66abc92e5c33fdef75c63dec81f90a3a Mon Sep 17 00:00:00 2001 From: Wouter van Oortmerssen Date: Thu, 4 Sep 2014 16:31:44 -0700 Subject: [PATCH] Implemented the file identifier functionality for Java. Also fixed flatc not outputting these identifiers for files compiled on the command-line. Bug: 16983987 Change-Id: I8b714cfea3a8e144fa52133f62b2f7eda6eb044a Tested: on Linux --- docs/html/md__compiler.html | 3 ++- docs/html/md__cpp_usage.html | 1 + docs/html/md__java_usage.html | 6 +++++- docs/source/JavaUsage.md | 18 ++++++++++++++++ include/flatbuffers/flatbuffers.h | 4 ++-- java/flatbuffers/Constants.java | 27 ++++++++++++++++++++++++ java/flatbuffers/FlatBufferBuilder.java | 17 ++++++++++----- java/flatbuffers/Table.java | 14 +++++++++--- src/idl_gen_java.cpp | 24 ++++++++++++++++++--- src/idl_parser.cpp | 3 ++- tests/JavaTest.java | 4 +++- tests/MyGame/Example/Monster.java | 3 +++ tests/MyGame/Example/Test.java | 1 + tests/MyGame/Example/Vec3.java | 1 + tests/monsterdata_test.bin | Bin 224 -> 240 bytes tests/monsterdata_test.json | 4 ++++ 16 files changed, 113 insertions(+), 17 deletions(-) create mode 100644 java/flatbuffers/Constants.java diff --git a/docs/html/md__compiler.html b/docs/html/md__compiler.html index 1f97fa3b7..d7bf180ec 100644 --- a/docs/html/md__compiler.html +++ b/docs/html/md__compiler.html @@ -64,7 +64,8 @@ $(document).ready(function(){initNavTree('md__compiler.html','');});
  • -b : If data is contained in this file, generate a filename.bin containing the binary flatbuffer.
  • -t : If data is contained in this file, generate a filename.json representing the data in the flatbuffer.
  • -o PATH : Output all generated files to PATH (either absolute, or relative to the current directory). If omitted, PATH will be the current directory. PATH should end in your systems path separator, e.g. / or \.
    • -
  • -S : Generate strict JSON (field names are enclosed in quotes). By default, no quotes are generated.
    • +
  • -S : Generate strict JSON (field names are enclosed in quotes). By default, no quotes are generated.
    • +
  • -P : Don't prefix enum values in generated C++ by their enum type.
    • diff --git a/docs/html/md__cpp_usage.html b/docs/html/md__cpp_usage.html index 9a74910f5..122f53b2e 100644 --- a/docs/html/md__cpp_usage.html +++ b/docs/html/md__cpp_usage.html @@ -104,6 +104,7 @@ assert(inv->Get(9) == 9);

    if ok is true, the buffer is safe to read.

    Besides untrusted data, this function may be useful to call in debug mode, as extra insurance against data being corrupted somewhere along the way.

    While verifying a buffer isn't "free", it is typically faster than a full traversal (since any scalar data is not actually touched), and since it may cause the buffer to be brought into cache before reading, the actual overhead may be even lower than expected.

    +

    In specialized cases where a denial of service attack is possible, the verifier has two additional constructor arguments that allow you to limit the nesting depth and total amount of tables the verifier may encounter before declaring the buffer malformed.

    Text & schema parsing

    Using binary buffers with the generated header provides a super low overhead use of FlatBuffer data. There are, however, times when you want to use text formats, for example because it interacts better with source control, or you want to give your users easy access to data.

    Another reason might be that you already have a lot of data in JSON format, or a tool that generates JSON, and if you can write a schema for it, this will provide you an easy way to use that data directly.

    diff --git a/docs/html/md__java_usage.html b/docs/html/md__java_usage.html index 99b923048..b9b9ee023 100644 --- a/docs/html/md__java_usage.html +++ b/docs/html/md__java_usage.html @@ -63,7 +63,9 @@ Vec3 pos = monster.pos();

    Sadly the string accessors currently always create a new string when accessed, since FlatBuffer's UTF-8 strings can't be read in-place by Java.

    Vector access is also a bit different from C++: you pass an extra index to the vector field accessor. Then a second method with the same name suffixed by Length let's you know the number of elements you can access: 

    for (int i = 0; i < monster.inventoryLength(); i++)
     monster.inventory(i); // do something here
-

    You can also construct these buffers in Java using the static methods found in the generated code, and the FlatBufferBuilder class: 

    FlatBufferBuilder fbb = new FlatBufferBuilder();
+

    If you specified a file_indentifier in the schema, you can query if the buffer is of the desired type before accessing it using: 

    if (Monster.MonsterBufferHasIdentifier(bb, start)) ...
+

    Buffer construction in Java

    +

    You can also construct these buffers in Java using the static methods found in the generated code, and the FlatBufferBuilder class: 

    FlatBufferBuilder fbb = new FlatBufferBuilder();

    Create strings: 

    int str = fbb.createString("MyMonster");

    Create a table with a struct contained therein: 

    Monster.startMonster(fbb);
 Monster.addPos(fbb, Vec3.createVec3(fbb, 1.0f, 2.0f, 3.0f, 3.0, (byte)4, (short)5, (byte)6));
@@ -83,6 +85,8 @@ for (byte i = 4; i >=0; i--) fbb.addByte(i);
 int inv = fbb.endVector();

    You can use the generated method startInventoryVector to conveniently call startVector with the right element size. You pass the number of elements you want to write. You write the elements backwards since the buffer is being constructed back to front.

    There are add functions for all the scalar types. You use addOffset for any previously constructed objects (such as other tables, strings, vectors). For structs, you use the appropriate create function in-line, as shown above in the Monster example.

    +

    To finish the buffer, call: 

    Monster.finishMonsterBuffer(fbb, mon);
+

    The buffer is now ready to be transmitted. It is contained in the ByteBuffer which you can obtain from fbb.dataBuffer(). Importantly, the valid data does not start from offset 0 in this buffer, but from fbb.dataStart() (this is because the data was built backwards in memory). It ends at fbb,capacity().

    Text Parsing

    There currently is no support for parsing text (Schema's and JSON) directly from Java, though you could use the C++ parser through JNI. Please see the C++ documentation for more on text parsing.

    diff --git a/docs/source/JavaUsage.md b/docs/source/JavaUsage.md index 4e2fe8bc8..15bd816dd 100755 --- a/docs/source/JavaUsage.md +++ b/docs/source/JavaUsage.md @@ -38,6 +38,14 @@ suffixed by `Length` let's you know the number of elements you can access: for (int i = 0; i < monster.inventoryLength(); i++) monster.inventory(i); // do something here +If you specified a file_indentifier in the schema, you can query if the +buffer is of the desired type before accessing it using: + + if (Monster.MonsterBufferHasIdentifier(bb, start)) ... + + +## Buffer construction in Java + You can also construct these buffers in Java using the static methods found in the generated code, and the FlatBufferBuilder class: @@ -91,6 +99,16 @@ any previously constructed objects (such as other tables, strings, vectors). For structs, you use the appropriate `create` function in-line, as shown above in the `Monster` example. +To finish the buffer, call: + + Monster.finishMonsterBuffer(fbb, mon); + +The buffer is now ready to be transmitted. It is contained in the `ByteBuffer` +which you can obtain from `fbb.dataBuffer()`. Importantly, the valid data does +not start from offset 0 in this buffer, but from `fbb.dataStart()` (this is +because the data was built backwards in memory). It ends at `fbb,capacity()`. + + ## Text Parsing There currently is no support for parsing text (Schema's and JSON) directly diff --git a/include/flatbuffers/flatbuffers.h b/include/flatbuffers/flatbuffers.h index bab6a4602..0ac4e81f0 100644 --- a/include/flatbuffers/flatbuffers.h +++ b/include/flatbuffers/flatbuffers.h @@ -665,8 +665,8 @@ template const T *GetRoot(const void *buf) { // Helper to see if the identifier in a buffer has the expected value. inline bool BufferHasIdentifier(const void *buf, const char *identifier) { - return strncmp(reinterpret_cast(buf) + 4, identifier, - FlatBufferBuilder::kFileIdentifierLength) == 0; + return strncmp(reinterpret_cast(buf) + sizeof(uoffset_t), + identifier, FlatBufferBuilder::kFileIdentifierLength) == 0; } // Helper class to verify the integrity of a FlatBuffer diff --git a/java/flatbuffers/Constants.java b/java/flatbuffers/Constants.java new file mode 100644 index 000000000..aeb22cc1f --- /dev/null +++ b/java/flatbuffers/Constants.java @@ -0,0 +1,27 @@ +/* + * Copyright 2014 Google Inc. All rights reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package flatbuffers; + +// Class that holds shared constants. + +public class Constants { + // Java doesn't seem to have these. + static final int SIZEOF_SHORT = 2; + static final int SIZEOF_INT = 4; + static final int FILE_IDENTIFIER_LENGTH = 4; +} + diff --git a/java/flatbuffers/FlatBufferBuilder.java b/java/flatbuffers/FlatBufferBuilder.java index 50adeb510..a1bc97487 100755 --- a/java/flatbuffers/FlatBufferBuilder.java +++ b/java/flatbuffers/FlatBufferBuilder.java @@ -25,7 +25,7 @@ import java.nio.charset.Charset; // Class that helps you build a FlatBuffer. // See the section "Use in Java" in the main FlatBuffers documentation. -public class FlatBufferBuilder { +public class FlatBufferBuilder extends Constants { ByteBuffer bb; // Where we construct the FlatBuffer. int space; // Remaining space in the ByteBuffer. static final Charset utf8charset = Charset.forName("UTF-8"); @@ -36,10 +36,6 @@ public class FlatBufferBuilder { int num_vtables = 0; // Number of entries in `vtables` in use. int vector_num_elems = 0; // For the current vector being built. - // Java doesn't seem to have these. - final int SIZEOF_SHORT = 2; - final int SIZEOF_INT = 4; - // Start with a buffer of size `initial_size`, then grow as required. public FlatBufferBuilder(int initial_size) { if (initial_size <= 0) initial_size = 1; @@ -251,6 +247,17 @@ public class FlatBufferBuilder { addOffset(root_table); } + public void finish(int root_table, String file_identifier) { + prep(minalign, SIZEOF_INT + FILE_IDENTIFIER_LENGTH); + if (file_identifier.length() != FILE_IDENTIFIER_LENGTH) + throw new AssertionError("FlatBuffers: file identifier must be length " + + FILE_IDENTIFIER_LENGTH); + for (int i = FILE_IDENTIFIER_LENGTH - 1; i >= 0; i--) { + addByte((byte)file_identifier.charAt(i)); + } + addOffset(root_table); + } + public ByteBuffer dataBuffer() { return bb; } // The FlatBuffer data doesn't start at offset 0 in the ByteBuffer: diff --git a/java/flatbuffers/Table.java b/java/flatbuffers/Table.java index f25188aef..7740e49fd 100755 --- a/java/flatbuffers/Table.java +++ b/java/flatbuffers/Table.java @@ -21,12 +21,10 @@ import java.nio.ByteBuffer; import java.nio.charset.Charset; // All tables in the generated code derive from this class, and add their own accessors. -public class Table { +public class Table extends Constants { protected int bb_pos; protected ByteBuffer bb; - final int SIZEOF_INT = 4; - // Look up a field in the vtable, return an offset into the object, or 0 if the field is not // present. protected int __offset(int vtable_offset) { @@ -75,4 +73,14 @@ public class Table { t.bb = bb; return t; } + + protected static boolean __has_identifier(ByteBuffer bb, int offset, String ident) { + if (ident.length() != FILE_IDENTIFIER_LENGTH) + throw new AssertionError("FlatBuffers: file identifier must be length " + + FILE_IDENTIFIER_LENGTH); + for (int i = 0; i < FILE_IDENTIFIER_LENGTH; i++) { + if (ident.charAt(i) != (char)bb.get(offset + SIZEOF_INT + i)) return false; + } + return true; + } } diff --git a/src/idl_gen_java.cpp b/src/idl_gen_java.cpp index 7c57ffbeb..8ee42006d 100755 --- a/src/idl_gen_java.cpp +++ b/src/idl_gen_java.cpp @@ -154,7 +154,8 @@ static void GenStructBody(const StructDef &struct_def, std::string *code_ptr, } } -static void GenStruct(StructDef &struct_def, std::string *code_ptr) { +static void GenStruct(const Parser &parser, StructDef &struct_def, + std::string *code_ptr) { if (struct_def.generated) return; std::string &code = *code_ptr; @@ -177,12 +178,21 @@ static void GenStruct(StructDef &struct_def, std::string *code_ptr) { code += "_bb.order(ByteOrder.LITTLE_ENDIAN); "; code += "return (new " + struct_def.name; code += "()).__init(_bb.getInt(offset) + offset, _bb); }\n"; + if (parser.root_struct_def == &struct_def) { + if (parser.file_identifier_.length()) { + // Check if a buffer has the identifier. + code += " public static boolean " + struct_def.name; + code += "BufferHasIdentifier(ByteBuffer _bb, int offset) { return "; + code += "__has_identifier(_bb, offset, \"" + parser.file_identifier_; + code += "\"); }\n"; + } + } } // Generate the __init method that sets the field in a pre-existing // accessor object. This is to allow object reuse. code += " public " + struct_def.name; code += " __init(int _i, ByteBuffer _bb) "; - code += "{ bb_pos = _i; bb = _bb; return this; }\n"; + code += "{ bb_pos = _i; bb = _bb; return this; }\n\n"; for (auto it = struct_def.fields.vec.begin(); it != struct_def.fields.vec.end(); ++it) { @@ -321,6 +331,14 @@ static void GenStruct(StructDef &struct_def, std::string *code_ptr) { } code += " public static int end" + struct_def.name; code += "(FlatBufferBuilder builder) { return builder.endObject(); }\n"; + if (parser.root_struct_def == &struct_def) { + code += " public static void finish" + struct_def.name; + code += "Buffer(FlatBufferBuilder builder, int offset) { "; + code += "builder.finish(offset"; + if (parser.file_identifier_.length()) + code += ", \"" + parser.file_identifier_ + "\""; + code += "); }\n"; + } } code += "};\n\n"; } @@ -375,7 +393,7 @@ bool GenerateJava(const Parser &parser, for (auto it = parser.structs_.vec.begin(); it != parser.structs_.vec.end(); ++it) { std::string declcode; - GenStruct(**it, &declcode); + GenStruct(parser, **it, &declcode); if (!SaveClass(parser, **it, declcode, path, true)) return false; } diff --git a/src/idl_parser.cpp b/src/idl_parser.cpp index ed4476dba..af44a9324 100644 --- a/src/idl_parser.cpp +++ b/src/idl_parser.cpp @@ -914,7 +914,8 @@ bool Parser::Parse(const char *source, const char *filepath) { if (builder_.GetSize()) { Error("cannot have more than one json object in a file"); } - builder_.Finish(Offset(ParseTable(*root_struct_def))); + builder_.Finish(Offset
    (ParseTable(*root_struct_def)), + file_identifier_.length() ? file_identifier_.c_str() : nullptr); } else if (token_ == kTokenEnum) { ParseEnum(false); } else if (token_ == kTokenUnion) { diff --git a/tests/JavaTest.java b/tests/JavaTest.java index 862659e81..4747c44f3 100755 --- a/tests/JavaTest.java +++ b/tests/JavaTest.java @@ -85,7 +85,7 @@ class JavaTest { Monster.addTestarrayofstring(fbb, testArrayOfString); int mon = Monster.endMonster(fbb); - fbb.finish(mon); + Monster.finishMonsterBuffer(fbb, mon); // Write the result to a file for debugging purposes: // Note that the binaries are not necessarily identical, since the JSON @@ -113,6 +113,8 @@ class JavaTest { } static void TestBuffer(ByteBuffer bb, int start) { + TestEq(Monster.MonsterBufferHasIdentifier(bb, start), true); + Monster monster = Monster.getRootAsMonster(bb, start); TestEq(monster.hp(), (short)80); diff --git a/tests/MyGame/Example/Monster.java b/tests/MyGame/Example/Monster.java index 68da9d894..76f94760e 100755 --- a/tests/MyGame/Example/Monster.java +++ b/tests/MyGame/Example/Monster.java @@ -9,7 +9,9 @@ import flatbuffers.*; public class Monster extends Table { public static Monster getRootAsMonster(ByteBuffer _bb, int offset) { _bb.order(ByteOrder.LITTLE_ENDIAN); return (new Monster()).__init(_bb.getInt(offset) + offset, _bb); } + public static boolean MonsterBufferHasIdentifier(ByteBuffer _bb, int offset) { return __has_identifier(_bb, offset, "MONS"); } public Monster __init(int _i, ByteBuffer _bb) { bb_pos = _i; bb = _bb; return this; } + public Vec3 pos() { return pos(new Vec3()); } public Vec3 pos(Vec3 obj) { int o = __offset(4); return o != 0 ? obj.__init(o + bb_pos, bb) : null; } public short mana() { int o = __offset(6); return o != 0 ? bb.getShort(o + bb_pos) : 150; } @@ -57,5 +59,6 @@ public class Monster extends Table { public static void startTestnestedflatbufferVector(FlatBufferBuilder builder, int numElems) { builder.startVector(1, numElems, 1); } public static void addTestempty(FlatBufferBuilder builder, int testemptyOffset) { builder.addOffset(14, testemptyOffset, 0); } public static int endMonster(FlatBufferBuilder builder) { return builder.endObject(); } + public static void finishMonsterBuffer(FlatBufferBuilder builder, int offset) { builder.finish(offset, "MONS"); } }; diff --git a/tests/MyGame/Example/Test.java b/tests/MyGame/Example/Test.java index 93261f967..fde24e655 100755 --- a/tests/MyGame/Example/Test.java +++ b/tests/MyGame/Example/Test.java @@ -9,6 +9,7 @@ import flatbuffers.*; public class Test extends Struct { public Test __init(int _i, ByteBuffer _bb) { bb_pos = _i; bb = _bb; return this; } + public short a() { return bb.getShort(bb_pos + 0); } public byte b() { return bb.get(bb_pos + 2); } diff --git a/tests/MyGame/Example/Vec3.java b/tests/MyGame/Example/Vec3.java index 652cba946..be5c66b34 100755 --- a/tests/MyGame/Example/Vec3.java +++ b/tests/MyGame/Example/Vec3.java @@ -9,6 +9,7 @@ import flatbuffers.*; public class Vec3 extends Struct { public Vec3 __init(int _i, ByteBuffer _bb) { bb_pos = _i; bb = _bb; return this; } + public float x() { return bb.getFloat(bb_pos + 0); } public float y() { return bb.getFloat(bb_pos + 4); } public float z() { return bb.getFloat(bb_pos + 8); } diff --git a/tests/monsterdata_test.bin b/tests/monsterdata_test.bin index 952a96bef5f8c6a3e603f801aec9852d613f070f..4283d59c92bcaf9df88b9f1a117ab1e74004c4b4 100644 GIT binary patch delta 92 zcmaFB_<_;Vf`Ng-*WWJ~NFf0w1_uTXAfJsv2S^$KF)M=!g9U>PT#eF1O>NE^AU6ky U6DB5FGczzKO`IzaruHfW0Nxu5hX4Qo delta 76 zcmeys_<&JUg8>2*7+e@QfJ`<910ZPv#H