Merge changes from Antidote Refractor to integrate Gingko by deadlockcharlie · Pull Request #463 · AntidoteDB/antidote

deadlockcharlie · 2022-02-10T09:54:23Z

Removed Logging, Materialization and Caches
Object Cahge introduced in Gingko.

… for backwards compatibility with tests

This is to compare the original implementation with the new library.

…e_refractor

Complete refractor

dch · 2022-04-29T08:02:57Z

@ayushpandey8439 I'm going through this PR to see how we can most easily
merge it. To do that I need to have a clear understanding of its test
stability in each of the commits, to ensure I'm not breaking anything.

Right now, testing this PR on a vanilla amd64 Ubuntu 20.04, I don't get
tests passing on the PR itself. Can you share some background on how
you're testing this, and if there are any key commits where you'd
recommend using them as a solid base to validate tests in between current
Antidote master & this PR's tip?

environment

dch@u01 ~/s/a/pr463> uname -a
Linux u01.cabal5.net 5.13.0-1027-oracle #32~20.04.1-Ubuntu SMP Fri Apr 15 05:59:34 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
dch@u01 ~/s/a/pr463> lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 20.04.4 LTS
Release:        20.04
Codename:       focal

dch@u01 ~/s/a/pr463> erl -eval '{ok, Version} = file:read_file(filename:join([code:root_dir(), "releases", erlang:system_info(otp_release), "OTP_VERSION"])), io:fwrite(Version), halt().' -noshell
24.3.3

code

dch@u01 ~/s/a/pr463> ln -s ../../gingko _checkouts/gingko

dch@u01 ~/s/a/pr463> git log --oneline | head -4
c0aa0256 build: use relative path to gingko   <-- so I can use a local copy
189c55d8 Add transaction start record to the journal <--- your ayush/master
be18fe65 Merge pull request #1 from ayushpandey8439/complete_refractor
bff9838e Optimised Checkpoint Calculation ...

dch@u01 ~/s/a/gingko> git log --oneline | head -4
10e61ab Add start transaction record in the journal
18e4112 Optimised Checkpoint Calculation
298cb51 Merge remote-tracking branch 'origin/gingko-cache' into gingko-cache
747e5a4 Revert "Wait if logger busy"

rebar3 eunit

a basic eunit bails at this point:

rebar3 eunit
===> Verifying dependencies...
make: Entering directory '/home/dch/src/antidote/pr463/_build/default/lib/erlzmq/c_src'
make: Nothing to be done for 'all'.
make: Leaving directory '/home/dch/src/antidote/pr463/_build/default/lib/erlzmq/c_src'
===> Analyzing applications...
===> Compiling gingko
===> Compiling ranch
===> Compiling accept
===> Compiling elli_prometheus
===> Compiling antidote_stats
===> Compiling antidote_pb_codec
===> Compiling antidotec_pb
===> Compiling erlzmq
===> Compiling poolboy
===> Compiling riak_core
===> "/home/dch/src/antidote/pr463/_build/test/checkouts/gingko/ebin/gingko.app" is missing kernel from applications list
===> "/home/dch/src/antidote/pr463/_build/test/checkouts/gingko/ebin/gingko.app" is missing stdlib from applications list
===> Compiling c_src/erlzmq_nif.c
===> Compiling c_src/vector.c
===> Analyzing applications...
===> Compiling antidote
===> Performing EUnit tests...
...=CRASH REPORT==== 29-Apr-2022::07:56:28.373870 ===
  crasher:
    initial call: meck_proc:init/1
    pid: <0.1277.0>
    registered_name: []
    exception error: {undefined_module,logging_vnode}
      in function  meck_proc:resolve_can_expect/3 (/home/dch/src/antidote/pr463/_build/test/lib/meck/src/meck_proc.erl, line 456)
      in call from meck_proc:init/1 (/home/dch/src/antidote/pr463/_build/test/lib/meck/src/meck_proc.erl, line 227)
      in call from gen_server:init_it/2 (gen_server.erl, line 423)
      in call from gen_server:init_it/6 (gen_server.erl, line 390)
    ancestors: [<0.1213.0>]
    message_queue_len: 0
    messages: []
    links: [<0.1213.0>]
    dictionary: []
    trap_exit: false
    status: running
    heap_size: 987
    stack_size: 29
    reductions: 322
  neighbours:
    neighbour:
      pid: <0.1218.0>
      registered_name: zmq_context
      initial call: zmq_context:init/1
      current_function: {gen_server,loop,7}
      ancestors: [<0.1213.0>]
      message_queue_len: 0
      links: [<0.1213.0>]
      trap_exit: false
      status: waiting
      heap_size: 233
      stack_size: 12
      reductions: 245
      current_stacktrace: [{gen_server,loop,7,[{file,"gen_server.erl"},{line,443}]},
                  {proc_lib,init_p_do_apply,3,
                            [{file,"proc_lib.erl"},{line,226}]}]
    neighbour:
      pid: <0.1213.0>
      registered_name: []
      initial_call: {erlang,apply,2}
      current_function: {code_server,call,1}
      ancestors: []
      message_queue_len: 0
      links: [<0.1218.0>,<0.1270.0>,<0.1277.0>,<0.1229.0>,<0.1212.0>]
      trap_exit: false
      status: waiting
      heap_size: 6772
      stack_size: 68
      reductions: 19179
      current_stacktrace: [{code_server,call,1,[{file,"code_server.erl"},{line,139}]},
                  {error_handler,undefined_function,3,
                                 [{file,"error_handler.erl"},{line,40}]},
                  {eunit_data,get_module_tests,1,
                              [{file,"eunit_data.erl"},{line,560}]},
                  {eunit_data,get_module_tests_1,2,
                              [{file,"eunit_data.erl"},{line,581}]},
                  {eunit_data,get_module_tests,1,
                              [{file,"eunit_data.erl"},{line,562}]},
                  {eunit_data,parse,1,[{file,"eunit_data.erl"},{line,341}]},
                  {eunit_data,next,1,[{file,"eunit_data.erl"},{line,175}]},
                  {eunit_data,iter_next,1,
                              [{file,"eunit_data.erl"},{line,135}]}]

Pending:
  module 'inter_dc_sub_buf'
    %% Unknown error: {abort,
                   {setup_failed,
                       {error,
                           {{undefined_module,logging_vnode},
                            [{meck_proc,resolve_can_expect,3,
                                 [{file,
                                      "/home/dch/src/antidote/pr463/_build/test/lib/meck/src/meck_proc.erl"},
                                  {line,456}]},
                             {meck_proc,init,1,
                                 [{file,
                                      "/home/dch/src/antidote/pr463/_build/test/lib/meck/src/meck_proc.erl"},
                                  {line,227}]},
                             {gen_server,init_it,2,
                                 [{file,"gen_server.erl"},{line,423}]},
                             {gen_server,init_it,6,
                                 [{file,"gen_server.erl"},{line,390}]},
                             {proc_lib,init_p_do_apply,3,
                                 [{file,"proc_lib.erl"},{line,226}]}]},
                           [{meck_proc,start,
                                [logging_vnode,[]],
                                [{file,
                                     "/home/dch/src/antidote/pr463/_build/test/lib/meck/src/meck_proc.erl"},
                                 {line,93}]},
                            {inter_dc_sub_buf,test_init,0,
                                [{file,
                                     "/home/dch/src/antidote/pr463/src/inter_dc/inter_dc_sub_buf.erl"},
                                 {line,245}]}]}}}
  undefined
    %% Related process exited with reason: {{undefined_module,logging_vnode},
                                     [{meck_proc,resolve_can_expect,3,
                                       [{file,
                                         "/home/dch/src/antidote/pr463/_build/test/lib/meck/src/meck_proc.erl"},
                                        {line,456}]},
                                      {meck_proc,init,1,
                                       [{file,
                                         "/home/dch/src/antidote/pr463/_build/test/lib/meck/src/meck_proc.erl"},
                                        {line,227}]},
                                      {gen_server,init_it,2,
                                       [{file,"gen_server.erl"},{line,423}]},
                                      {gen_server,init_it,6,
                                       [{file,"gen_server.erl"},{line,390}]},
                                      {proc_lib,init_p_do_apply,3,
                                       [{file,"proc_lib.erl"},{line,226}]}]}


Finished in ? seconds
7 tests, 0 failures, 4 cancelled
=ERROR REPORT==== 29-Apr-2022::07:56:28.960395 ===
** Generic server inter_dc_query_response_meck terminating
** Last message in was {'EXIT',<0.1213.0>,
                           {{undefined_module,logging_vnode},
                            [{meck_proc,resolve_can_expect,3,
                                 [{file,
                                      "/home/dch/src/antidote/pr463/_build/test/lib/meck/src/meck_proc.erl"},
                                  {line,456}]},
                             {meck_proc,init,1,
                                 [{file,
                                      "/home/dch/src/antidote/pr463/_build/test/lib/meck/src/meck_proc.erl"},
                                  {line,227}]},
                             {gen_server,init_it,2,
                                 [{file,"gen_server.erl"},{line,423}]},
                             {gen_server,init_it,6,
                                 [{file,"gen_server.erl"},{line,390}]},
                             {proc_lib,init_p_do_apply,3,
                                 [{file,"proc_lib.erl"},{line,226}]}]}}
** When Server state == {state,inter_dc_query_response,
                         [{start_link,1},
                          {get_entries,2},
                          {request_permissions,2},
                          {init,1},
                          {handle_cast,2},
                          {handle_call,3},
                          {handle_info,2},
                          {terminate,2},
                          {code_change,3},
                          {generate_server_name,1}],
                         {dict,2,16,16,8,80,48,
                          {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
                          {{[],[],[],[],[],[],[],
                            [[{request_permissions,2}|
                              {{request_permissions,2},
                               [{{args_matcher,
                                  ['_','_'],
                                  #Ref<0.3386037366.3149004801.43992>,false},
                                 {meck_exec,
                                  #Fun<inter_dc_query_router.5.94840950>}}]}],
                             [{get_entries,2}|
                              {{get_entries,2},
                               [{{args_matcher,
                                  ['_','_'],
                                  #Ref<0.3386037366.3149004802.17425>,false},
                                 {meck_exec,
                                  #Fun<inter_dc_query_router.1.94840950>}}]}]],
                            [],[],[],[],[],[],[],[]}}},
                         true,
                         [{<0.1253.0>,
                           {inter_dc_query_response,get_entries,
                            [<<131,104,4,100,0,8,114,101,97,100,95,108,111,103,
                               97,0,97,1,97,4>>,
                             {inter_dc_query_state,2,
                              <<0,228,60,152,106>>,
                              <<0,1>>,
                              <0.1253.0>}]},
                           ok},
                          {<0.1246.0>,
                           {inter_dc_query_response,request_permissions,
                            [<<131,104,5,100,0,19,114,101,113,117,101,115,116,
                               95,112,101,114,109,105,115,115,105,111,110,115,
                               104,2,100,0,8,116,114,97,110,115,102,101,114,
                               104,3,107,0,5,104,101,108,108,111,97,0,100,0,4,
                               100,99,105,100,97,0,100,0,4,100,99,105,100,97,0>>,
                             {inter_dc_query_state,6,
                              <<0,107,139,69,104>>,
                              <<0,1>>,
                              <0.1246.0>}]},
                           ok}],
                         {{"/home/dch/src/antidote/pr463/_build/test/lib/antidote/ebin/inter_dc_query_response.beam",
                           "/home/dch/src/antidote/pr463/inter_dc_query_response.77879.coverdata",
                           [debug_info,no_spawn_compiler_process,
                            {d,'EUNIT'},
                            nowarn_export_all,
                            {i,"include"},
                            {d,'TEST'},
                            no_inline_list_funcs,
                            {i,
                             "/home/dch/src/antidote/pr463/_build/test/lib/antidote/src"},
                            {i,
                             "/home/dch/src/antidote/pr463/_build/test/lib/antidote/test"},
                            {i,
                             "/home/dch/src/antidote/pr463/_build/test/lib/antidote/include"},
                            {i,
                             "/home/dch/src/antidote/pr463/_build/test/lib/antidote"}]},
                          <<70,79,82,49,0,0,40,252,66,69,65,77,65,116,85,56,0,
                            0,3,62,0,0,0,68,37,105,110,116,101,114,95,100,99,
...                            101,114,108>>},
                         false,false,false,undefined,[]}
** Reason for termination ==
** {{undefined_module,logging_vnode},
    [{meck_proc,resolve_can_expect,3,
                [{file,"/home/dch/src/antidote/pr463/_build/test/lib/meck/src/meck_proc.erl"},
                 {line,456}]},
     {meck_proc,init,1,
                [{file,"/home/dch/src/antidote/pr463/_build/test/lib/meck/src/meck_proc.erl"},
                 {line,227}]},
     {gen_server,init_it,2,[{file,"gen_server.erl"},{line,423}]},
     {gen_server,init_it,6,[{file,"gen_server.erl"},{line,390}]},
     {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}
...
      in function  gen_server:decode_msg/9 (gen_server.erl, line 481)
    ancestors: [<0.1213.0>]
    message_queue_len: 0
    messages: []
    links: []
    dictionary: []
    trap_exit: true
    status: running
    heap_size: 514838
    stack_size: 29
    reductions: 1996426
  neighbours:

===> Error running tests

define-null · 2022-04-29T15:44:59Z

I suggest to get rid of checkouts directory:

diff --git a/_checkouts/gingko b/_checkouts/gingko
deleted file mode 120000
index 71193e12..00000000
--- a/_checkouts/gingko
+++ /dev/null
@@ -1 +0,0 @@
-/Users/pandey/work/gingko/
\ No newline at end of file
diff --git a/rebar.config b/rebar.config
index 95865895..dc77978f 100644
--- a/rebar.config
+++ b/rebar.config
@@ -9,7 +9,8 @@
     antidote_pb_codec,
     antidotec_pb,
     vectorclock,
-    gingko,
+    {gingko, {git, "https://github.com/ayushpandey8439/gingko.git", {ref, "10e61abb02"}}},
+
     %% efficient inter-dc messaging
     {erlzmq, {git, "https://github.com/zeromq/erlzmq2", {ref, "573d583"}}},

diff --git a/rebar.lock b/rebar.lock
index c238cc30..514e96dc 100644
--- a/rebar.lock
+++ b/rebar.lock
@@ -13,6 +13,10 @@
   {git,"https://github.com/zeromq/erlzmq2",
        {ref,"573d583930c4b1134e504bec83926d188112b401"}},
   0},
+ {<<"gingko">>,
+  {git,"https://github.com/ayushpandey8439/gingko.git",
+       {ref,"10e61abb022ac2ef42851ec32e9150b34fb520fd"}},
+  0},
  {<<"poolboy">>,{pkg,<<"poolboy">>,<<"1.5.2">>},1},
  {<<"prometheus">>,{pkg,<<"prometheus">>,<<"4.6.0">>},1},
  {<<"prometheus_process_collector">>,

deadlockcharlie · 2022-04-29T19:46:59Z

Yes, That'd make things much easier. Especially if code in gingko is not being actively changed.

define-null · 2022-05-02T11:32:08Z

Starting from 9695e13 all calls to logging_vnode should go to gingko_vnode, as that is a new replacement.

deadlockcharlie added 30 commits September 6, 2021 16:17

Simplify the Update operation and remove auxilary function introduced…

49a73e4

… for backwards compatibility with tests

Replace function names asyn -> async

9abc31e

Replace Cache disctionary with multilevel ETS Cache

fa90fb8

Save state while coding (unstable changes).

d7c6054

Revert state before breaking changes

35b4f95

Start gingo as a child application in Antidote

8765bf5

Start gingko as a child application in Antidote

9695e13

Get_version routed from antidote to Gingko backend.

24de08c

Read routed from Antidote to Gingko with Journal materialization works.

dbfd5c1

Save state

4585137

Save State

aaa79d4

Rewrite the transaction coordinator and reduce code complexity

246d500

Detangle prepare and commit triggers and simplify redundant cases

0f9edf9

Update caches when a new transaction prepares or commits

9b40f56

Clockless read and update routed through the coordinator

f149e6c

Fix response sending for reads with clocks.

f16d64e

fix coordinator state for reads and updates

34d5464

Setup Single Dc tests

ba908fb

Single DC tests passing

4870f2a

Save state

87d8e35

Implement the checkpoint store

1be97bf

Send Key type in the writeset of the partition

010499a

Refractor tests

f073c87

Add key in update for saving log end pointer

feceff6

Remove checkpointing trigger

76a57d8

Send Transaction ID for checkpointing

df18d39

Add checkouts to gitignore

c134a64

Add PB Back into antidote

cc6539d

This is to compare the original implementation with the new library.

Include Buckets in Operations

0fbd917

fix error with multiple object calls

92e2174

Ayush Pandey and others added 5 commits January 20, 2022 08:58

Merge remote-tracking branch 'origin/complete_refractor' into complet…

f7c4bce

…e_refractor

Save state

63d5144

Optimised Checkpoint Calculation

bff9838

Merge pull request #1 from ayushpandey8439/complete_refractor

be18fe6

Complete refractor

Add transaction start record to the journal

189c55d

dch mentioned this pull request May 5, 2022

ginkgo ETS cache electric-sql/vaxine#11

Open

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Merge changes from Antidote Refractor to integrate Gingko#463

Merge changes from Antidote Refractor to integrate Gingko#463
deadlockcharlie wants to merge 35 commits intoAntidoteDB:masterfrom
deadlockcharlie:master

deadlockcharlie commented Feb 10, 2022

Uh oh!

dch commented Apr 29, 2022

Uh oh!

define-null commented Apr 29, 2022 •

edited

Loading

Uh oh!

deadlockcharlie commented Apr 29, 2022 •

edited

Loading

Uh oh!

define-null commented May 2, 2022 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

deadlockcharlie commented Feb 10, 2022

Uh oh!

dch commented Apr 29, 2022

environment

code

rebar3 eunit

Uh oh!

define-null commented Apr 29, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

deadlockcharlie commented Apr 29, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

define-null commented May 2, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

define-null commented Apr 29, 2022 •

edited

Loading

deadlockcharlie commented Apr 29, 2022 •

edited

Loading

define-null commented May 2, 2022 •

edited

Loading