added test diagrams, gantt chart, sent to Ning
This commit is contained in:
parent
3a1dacd6da
commit
19d40c5a4c
BIN
GanttChart.ods
Normal file
BIN
GanttChart.ods
Normal file
Binary file not shown.
@ -245,10 +245,10 @@ LiveScan3D
|
|||||||
\noun default
|
\noun default
|
||||||
suite is presented, facilitating the acquisition and presentation of multiple
|
suite is presented, facilitating the acquisition and presentation of multiple
|
||||||
scenes simultaneously.
|
scenes simultaneously.
|
||||||
In doing so the application suite already strengthened by it's domain-agnostic
|
In doing so the application suite already strengthened by its domain-agnostic
|
||||||
nature can be seen to be further generalised allowing conference call-like
|
nature can be seen to be further generalised allowing conference call-like
|
||||||
experiences.
|
experiences.
|
||||||
Developments were made to the server display component allowing this concurrent
|
Developments were made to the server display component to allow this concurrent
|
||||||
presentation and to the network layer of the suite in order to differentiate
|
presentation and to the network layer of the suite in order to differentiate
|
||||||
sources.
|
sources.
|
||||||
Extensions to the
|
Extensions to the
|
||||||
@ -460,11 +460,11 @@ This project aims to extend this suite to support multi-source holoportation,
|
|||||||
|
|
||||||
\begin_layout Standard
|
\begin_layout Standard
|
||||||
One application would be support for experiences akin to conference-calls
|
One application would be support for experiences akin to conference-calls
|
||||||
with multiple actors capturing their own environments for composition at
|
with multiple actors capturing their separate environments for composition
|
||||||
the server.
|
at the server.
|
||||||
This could be seen to be applicable both to productivity software similar
|
This could be seen to apply both to productivity software similar to existing
|
||||||
to existing conference-call software and to entertainment experiences with
|
conference-call software and entertainment experiences with multiple locations
|
||||||
multiple locations combined for consumption by the public.
|
combined for consumption by the public.
|
||||||
|
|
||||||
\end_layout
|
\end_layout
|
||||||
|
|
||||||
@ -594,7 +594,7 @@ status open
|
|||||||
\begin_inset Graphics
|
\begin_inset Graphics
|
||||||
filename ../media/premise.png
|
filename ../media/premise.png
|
||||||
lyxscale 30
|
lyxscale 30
|
||||||
width 70col%
|
width 60col%
|
||||||
|
|
||||||
\end_inset
|
\end_inset
|
||||||
|
|
||||||
@ -637,17 +637,6 @@ Objectives
|
|||||||
\begin_layout Standard
|
\begin_layout Standard
|
||||||
In order to achieve the goal of multi-source holoportation the following
|
In order to achieve the goal of multi-source holoportation the following
|
||||||
key objectives must be achieved,
|
key objectives must be achieved,
|
||||||
\begin_inset Flex TODO Note (inline)
|
|
||||||
status open
|
|
||||||
|
|
||||||
\begin_layout Plain Layout
|
|
||||||
Need more explanation of the objectives below, in particular what is the
|
|
||||||
technical challenge
|
|
||||||
\end_layout
|
|
||||||
|
|
||||||
\end_inset
|
|
||||||
|
|
||||||
|
|
||||||
\end_layout
|
\end_layout
|
||||||
|
|
||||||
\begin_layout Enumerate
|
\begin_layout Enumerate
|
||||||
@ -658,11 +647,15 @@ Introduce a source ID in network communications to identify each frame of
|
|||||||
\begin_layout Enumerate
|
\begin_layout Enumerate
|
||||||
Migrate the server from a single-source streaming node to a manager of multiple
|
Migrate the server from a single-source streaming node to a manager of multiple
|
||||||
scenes including a method to synchronise sources.
|
scenes including a method to synchronise sources.
|
||||||
|
This includes defining methods of identifying sources that are no longer
|
||||||
|
actively transmitting frames to remove them from presentation.
|
||||||
\end_layout
|
\end_layout
|
||||||
|
|
||||||
\begin_layout Enumerate
|
\begin_layout Enumerate
|
||||||
Extend the native viewfinder of the server in order to separately render
|
Extend the native display viewfinder of the server in order to separately
|
||||||
each connected source.
|
render each connected source.
|
||||||
|
Allow individual placement of holograms using keyboard controls.
|
||||||
|
Include automatically coherent placement of newly connected sources.
|
||||||
\end_layout
|
\end_layout
|
||||||
|
|
||||||
\begin_layout Enumerate
|
\begin_layout Enumerate
|
||||||
@ -672,6 +665,7 @@ Android
|
|||||||
\noun default
|
\noun default
|
||||||
AR application to separately present each source of footage and allow individua
|
AR application to separately present each source of footage and allow individua
|
||||||
l control.
|
l control.
|
||||||
|
This would represent multi-source developments to an AR presentation layer.
|
||||||
\end_layout
|
\end_layout
|
||||||
|
|
||||||
\begin_layout Standard
|
\begin_layout Standard
|
||||||
@ -686,6 +680,13 @@ noprefix "false"
|
|||||||
\end_inset
|
\end_inset
|
||||||
|
|
||||||
.
|
.
|
||||||
|
The greatest challenge to a seamless multi-source experience is that of
|
||||||
|
the previously mentioned synchronisation.
|
||||||
|
While methods to achieve temporally matched presentation are investigated,
|
||||||
|
these have the potential to significantly increase complexity in the processing
|
||||||
|
pipeline.
|
||||||
|
This makes the design of these systems critical in maintaining efficient
|
||||||
|
real-time processing.
|
||||||
\end_layout
|
\end_layout
|
||||||
|
|
||||||
\begin_layout Standard
|
\begin_layout Standard
|
||||||
@ -766,8 +767,8 @@ e capabilities and some elements of the mobile AR display.
|
|||||||
\end_layout
|
\end_layout
|
||||||
|
|
||||||
\begin_layout Enumerate
|
\begin_layout Enumerate
|
||||||
Alter the AR environment of the mobile application in order to allow deployment
|
Migrate the AR environment of the mobile application in order to allow deploymen
|
||||||
to both
|
t to both
|
||||||
\noun on
|
\noun on
|
||||||
Android
|
Android
|
||||||
\noun default
|
\noun default
|
||||||
@ -779,6 +780,13 @@ Implement a dynamic system of frame rate throttling to investigate the effect
|
|||||||
on effective display latency.
|
on effective display latency.
|
||||||
\end_layout
|
\end_layout
|
||||||
|
|
||||||
|
\begin_layout Standard
|
||||||
|
The biggest challenge to the mobile AR migration is the scope of changes
|
||||||
|
being made to the codebase, changing the library used to create the AR
|
||||||
|
environment.
|
||||||
|
This is a significant change and poses many opportunities to cause issues.
|
||||||
|
\end_layout
|
||||||
|
|
||||||
\begin_layout Subsection
|
\begin_layout Subsection
|
||||||
Key Technical Achievements
|
Key Technical Achievements
|
||||||
\end_layout
|
\end_layout
|
||||||
@ -788,20 +796,6 @@ This section summarises the design and implementation of the developments
|
|||||||
made within this project.
|
made within this project.
|
||||||
\end_layout
|
\end_layout
|
||||||
|
|
||||||
\begin_layout Standard
|
|
||||||
\begin_inset Flex TODO Note (Margin)
|
|
||||||
status open
|
|
||||||
|
|
||||||
\begin_layout Plain Layout
|
|
||||||
minimum two pages to highlight/summarize what you have done, including your
|
|
||||||
design, implementation, testing strategy and key observations.
|
|
||||||
\end_layout
|
|
||||||
|
|
||||||
\end_inset
|
|
||||||
|
|
||||||
|
|
||||||
\end_layout
|
|
||||||
|
|
||||||
\begin_layout Subsubsection
|
\begin_layout Subsubsection
|
||||||
Multi-source LiveScan
|
Multi-source LiveScan
|
||||||
\end_layout
|
\end_layout
|
||||||
@ -888,12 +882,12 @@ Kinect
|
|||||||
\begin_layout Standard
|
\begin_layout Standard
|
||||||
The mobile AR application was updated in order to allow the simultaneous
|
The mobile AR application was updated in order to allow the simultaneous
|
||||||
display of multiple sources.
|
display of multiple sources.
|
||||||
Utilising the strengths of
|
Utilising strengths of
|
||||||
\noun on
|
\noun on
|
||||||
Unity
|
Unity
|
||||||
\noun default
|
\noun default
|
||||||
development, this did not require a significant redesign of the written
|
development, this did not require a significant redesign of the written
|
||||||
code, instead the hierarchy of game objects and components could be altered.
|
code, instead, the hierarchy of game objects and components could be altered.
|
||||||
The
|
The
|
||||||
\noun on
|
\noun on
|
||||||
PointCloudRenderer
|
PointCloudRenderer
|
||||||
@ -1167,11 +1161,7 @@ ts outlined.
|
|||||||
\begin_inset Newline newline
|
\begin_inset Newline newline
|
||||||
\end_inset
|
\end_inset
|
||||||
|
|
||||||
In support of ongoing research into the
|
Section
|
||||||
\noun on
|
|
||||||
LiveScan
|
|
||||||
\noun default
|
|
||||||
suite and it’s network behaviour, section
|
|
||||||
\begin_inset CommandInset ref
|
\begin_inset CommandInset ref
|
||||||
LatexCommand vref
|
LatexCommand vref
|
||||||
reference "sec:Frame-Rate-Throttling"
|
reference "sec:Frame-Rate-Throttling"
|
||||||
@ -1246,6 +1236,16 @@ noprefix "false"
|
|||||||
In the appendices,
|
In the appendices,
|
||||||
\begin_inset CommandInset ref
|
\begin_inset CommandInset ref
|
||||||
LatexCommand ref
|
LatexCommand ref
|
||||||
|
reference "sec:Gantt-Chart"
|
||||||
|
plural "false"
|
||||||
|
caps "false"
|
||||||
|
noprefix "false"
|
||||||
|
|
||||||
|
\end_inset
|
||||||
|
|
||||||
|
presents a Gantt chart for the project while
|
||||||
|
\begin_inset CommandInset ref
|
||||||
|
LatexCommand ref
|
||||||
reference "sec:Server-UI-Additions"
|
reference "sec:Server-UI-Additions"
|
||||||
plural "false"
|
plural "false"
|
||||||
caps "false"
|
caps "false"
|
||||||
@ -1500,20 +1500,10 @@ literal "false"
|
|||||||
\begin_layout Standard
|
\begin_layout Standard
|
||||||
A visual hull defines a 3D representation of an object constructed through
|
A visual hull defines a 3D representation of an object constructed through
|
||||||
the volume intersection of multiple 2D silhouettes.
|
the volume intersection of multiple 2D silhouettes.
|
||||||
With knowledge of the relative positions of each view point, corresponding
|
With knowledge of the relative positions of each viewpoint, corresponding
|
||||||
silhouettes of an object can be triangulated to form a 3D object.
|
silhouettes of an object can be triangulated to form a 3D object.
|
||||||
In doing so depth information lost in projection to a 2D image can be reconstru
|
In doing so depth information lost in projection to a 2D image can be reconstru
|
||||||
cted.
|
cted.
|
||||||
\begin_inset Flex TODO Note (Margin)
|
|
||||||
status open
|
|
||||||
|
|
||||||
\begin_layout Plain Layout
|
|
||||||
diagram?
|
|
||||||
\end_layout
|
|
||||||
|
|
||||||
\end_inset
|
|
||||||
|
|
||||||
|
|
||||||
\end_layout
|
\end_layout
|
||||||
|
|
||||||
\begin_layout Standard
|
\begin_layout Standard
|
||||||
@ -1593,7 +1583,7 @@ literal "false"
|
|||||||
|
|
||||||
\begin_layout Standard
|
\begin_layout Standard
|
||||||
The strength of these techniques draws from data gathered from tightly calibrate
|
The strength of these techniques draws from data gathered from tightly calibrate
|
||||||
d camera viewpoints, however when considering consumer-grade applications
|
d camera viewpoints, however, when considering consumer-grade applications
|
||||||
this is typically not available.
|
this is typically not available.
|
||||||
The use of infra-red structured light and
|
The use of infra-red structured light and
|
||||||
\emph on
|
\emph on
|
||||||
@ -1705,7 +1695,7 @@ literal "false"
|
|||||||
|
|
||||||
\begin_layout Standard
|
\begin_layout Standard
|
||||||
Here 3D conference calling of the type described in the introduction without
|
Here 3D conference calling of the type described in the introduction without
|
||||||
AR or VR applications is presented, instead users watch a composite conference
|
AR or VR applications is presented, instead, users watch a composite conference
|
||||||
space on a screen with all participants rendered within.
|
space on a screen with all participants rendered within.
|
||||||
|
|
||||||
\begin_inset Note Comment
|
\begin_inset Note Comment
|
||||||
@ -2234,7 +2224,7 @@ Android
|
|||||||
|
|
||||||
\begin_layout Standard
|
\begin_layout Standard
|
||||||
The advancement of mobile AR bolstered by the growing ubiquity of compatible
|
The advancement of mobile AR bolstered by the growing ubiquity of compatible
|
||||||
hardware has led it to becoming a rapidly growing and popular form of XR.
|
hardware has led it to become a rapidly growing and popular form of XR.
|
||||||
The introduction of OS-level SDK's in
|
The introduction of OS-level SDK's in
|
||||||
\noun on
|
\noun on
|
||||||
Google
|
Google
|
||||||
@ -2900,7 +2890,7 @@ ed.
|
|||||||
LiveScan3D
|
LiveScan3D
|
||||||
\noun default
|
\noun default
|
||||||
has two associated AR clients of which the most relevant to this project
|
has two associated AR clients of which the most relevant to this project
|
||||||
is the mobile AR client, subsequently the current SDKs facilitating AR
|
is the mobile AR client, subsequently, the current SDKs facilitating AR
|
||||||
development were presented and their relation to the
|
development were presented and their relation to the
|
||||||
\noun on
|
\noun on
|
||||||
Unity
|
Unity
|
||||||
@ -3603,7 +3593,7 @@ literal "false"
|
|||||||
\noun on
|
\noun on
|
||||||
Unity
|
Unity
|
||||||
\noun default
|
\noun default
|
||||||
and included buffers inline with the developments discussed in section
|
and included buffers in line with the developments discussed in section
|
||||||
|
|
||||||
\begin_inset CommandInset ref
|
\begin_inset CommandInset ref
|
||||||
LatexCommand ref
|
LatexCommand ref
|
||||||
@ -3959,8 +3949,8 @@ In order to maintain a strength of the suite in it's agnostic behaviour
|
|||||||
to different display methods, it was made critical to the design of the
|
to different display methods, it was made critical to the design of the
|
||||||
OpenGL extension that spatial processing would be specific to this display
|
OpenGL extension that spatial processing would be specific to this display
|
||||||
only.
|
only.
|
||||||
Mobile AR applications will arrange and interact with holograms in a different
|
Mobile AR applications will arrange and interact with holograms differently,
|
||||||
way, as will head-mounted AR or VR applications.
|
as will head-mounted AR or VR applications.
|
||||||
In maintaining this separation of network and presentation layer each display
|
In maintaining this separation of network and presentation layer each display
|
||||||
can operate independently and no assumptions are made as to how a display
|
can operate independently and no assumptions are made as to how a display
|
||||||
method should handle each hologram.
|
method should handle each hologram.
|
||||||
@ -4447,7 +4437,7 @@ Each source is assigned a default transformation which can be overridden
|
|||||||
\end_layout
|
\end_layout
|
||||||
|
|
||||||
\begin_layout Standard
|
\begin_layout Standard
|
||||||
Sources are initially arranged in a circle around the origin in the center
|
Sources are initially arranged in a circle around the origin in the centre
|
||||||
of the space.
|
of the space.
|
||||||
This is done by retrieving a transformation from the
|
This is done by retrieving a transformation from the
|
||||||
\noun on
|
\noun on
|
||||||
@ -4708,8 +4698,8 @@ Within the
|
|||||||
\noun on
|
\noun on
|
||||||
LiveScan
|
LiveScan
|
||||||
\noun default
|
\noun default
|
||||||
suite two network communications required updating, those between the client
|
suite, two network communications required updating, those between the
|
||||||
and server and those between the server and user experiences.
|
client and server and those between the server and user experiences.
|
||||||
\end_layout
|
\end_layout
|
||||||
|
|
||||||
\begin_layout Standard
|
\begin_layout Standard
|
||||||
@ -4801,7 +4791,7 @@ Implementation
|
|||||||
\begin_layout Standard
|
\begin_layout Standard
|
||||||
An integer source ID was created to be included in communications throughout
|
An integer source ID was created to be included in communications throughout
|
||||||
the suite.
|
the suite.
|
||||||
For conciseness this was represented by a single byte during transmission
|
For conciseness, this was represented by a single byte during transmission
|
||||||
resulting in a range of possible source IDs between 0 and 127.
|
resulting in a range of possible source IDs between 0 and 127.
|
||||||
Considering the use case of a source representing a scene and the bandwidth
|
Considering the use case of a source representing a scene and the bandwidth
|
||||||
required for each, 128 possible concurrent sources was deemed sufficient.
|
required for each, 128 possible concurrent sources was deemed sufficient.
|
||||||
@ -4958,9 +4948,9 @@ ce.
|
|||||||
At a high level, sources can be considered synchronised if frames captured
|
At a high level, sources can be considered synchronised if frames captured
|
||||||
at the same time are displayed concurrently whether from the perspective
|
at the same time are displayed concurrently whether from the perspective
|
||||||
of the server or a connected user experience.
|
of the server or a connected user experience.
|
||||||
There are three main challenges to achieving this aim.
|
There are two main challenges to achieving this aim.
|
||||||
The following assumes that timestamps captured at both the client and server
|
The following assumes that timestamps captured at both the client and server
|
||||||
are synchronised using the included NTP functionality
|
are synchronised and therefore comparable using the included NTP functionality
|
||||||
\begin_inset CommandInset citation
|
\begin_inset CommandInset citation
|
||||||
LatexCommand cite
|
LatexCommand cite
|
||||||
key "livescan3d-buffers"
|
key "livescan3d-buffers"
|
||||||
@ -4990,16 +4980,6 @@ Although capturing frames at the same rate (30Hz), it is highly unlikely
|
|||||||
Secondly, during transmission a variable amount of latency will be introduced
|
Secondly, during transmission a variable amount of latency will be introduced
|
||||||
as a result of the network conditions and distance from client to server
|
as a result of the network conditions and distance from client to server
|
||||||
and from server to UE.
|
and from server to UE.
|
||||||
\begin_inset Flex TODO Note (Margin)
|
|
||||||
status open
|
|
||||||
|
|
||||||
\begin_layout Plain Layout
|
|
||||||
remember third?
|
|
||||||
\end_layout
|
|
||||||
|
|
||||||
\end_inset
|
|
||||||
|
|
||||||
|
|
||||||
\end_layout
|
\end_layout
|
||||||
|
|
||||||
\begin_layout Standard
|
\begin_layout Standard
|
||||||
@ -5062,7 +5042,7 @@ Active
|
|||||||
\emph default
|
\emph default
|
||||||
synchronisation includes efforts to tightly match frames during processing
|
synchronisation includes efforts to tightly match frames during processing
|
||||||
at the server in order to minimise the difference in latency since capture.
|
at the server in order to minimise the difference in latency since capture.
|
||||||
In practice this could be achieved at the point where frames are moved
|
In practice, this could be achieved at the point where frames are moved
|
||||||
from the various receiving buffers to the live buffer
|
from the various receiving buffers to the live buffer
|
||||||
\begin_inset Flex TODO Note (Margin)
|
\begin_inset Flex TODO Note (Margin)
|
||||||
status open
|
status open
|
||||||
@ -5309,7 +5289,7 @@ s making up a hologram.
|
|||||||
\end_layout
|
\end_layout
|
||||||
|
|
||||||
\begin_layout Standard
|
\begin_layout Standard
|
||||||
On each frame update the
|
On each frame update, the
|
||||||
\noun on
|
\noun on
|
||||||
PointCloudReceiver
|
PointCloudReceiver
|
||||||
\noun default
|
\noun default
|
||||||
@ -5458,7 +5438,7 @@ While this thread could be included in many places throughout the server
|
|||||||
|
|
||||||
\begin_layout Standard
|
\begin_layout Standard
|
||||||
As such the object-oriented programming concept of encapsulation was employed
|
As such the object-oriented programming concept of encapsulation was employed
|
||||||
to create a object responsible for containing live frames and checking
|
to create an object responsible for containing live frames and checking
|
||||||
for
|
for
|
||||||
\emph on
|
\emph on
|
||||||
stale
|
stale
|
||||||
@ -5661,7 +5641,7 @@ noprefix "false"
|
|||||||
\begin_layout Standard
|
\begin_layout Standard
|
||||||
The use of a source ID transmitted alongside each frame provides flexibility
|
The use of a source ID transmitted alongside each frame provides flexibility
|
||||||
in that processing can be decoupled from the network infrastructure itself.
|
in that processing can be decoupled from the network infrastructure itself.
|
||||||
Theoretically it would also allow clients to function as multiple sources
|
Theoretically, it would also allow clients to function as multiple sources
|
||||||
simultaneously, allowing the attachment of different source IDs to different
|
simultaneously, allowing the attachment of different source IDs to different
|
||||||
frames
|
frames
|
||||||
\begin_inset Foot
|
\begin_inset Foot
|
||||||
@ -5699,7 +5679,7 @@ An alternative would be to associate a socket with a source ID, negotiated
|
|||||||
The server display's control scheme could be more intuitive as the directions
|
The server display's control scheme could be more intuitive as the directions
|
||||||
of movement are in relation to the fixed axes of the display space instead
|
of movement are in relation to the fixed axes of the display space instead
|
||||||
of the view of the camera.
|
of the view of the camera.
|
||||||
In practice this means that when translating objects the pose of the camera
|
In practice, this means that when translating objects the pose of the camera
|
||||||
with respect to the space must first be considered in order to identify
|
with respect to the space must first be considered in order to identify
|
||||||
which direction the object will be moved.
|
which direction the object will be moved.
|
||||||
\end_layout
|
\end_layout
|
||||||
@ -5867,7 +5847,7 @@ Unity ARFoundation
|
|||||||
\end_layout
|
\end_layout
|
||||||
|
|
||||||
\begin_layout Standard
|
\begin_layout Standard
|
||||||
Following these issues a different avenue was pursued in
|
Following these issues, a different avenue was pursued in
|
||||||
\noun on
|
\noun on
|
||||||
Unity
|
Unity
|
||||||
\noun default
|
\noun default
|
||||||
@ -5933,11 +5913,11 @@ Both libraries use a session component to manage the life cycle of the AR
|
|||||||
Unity
|
Unity
|
||||||
\noun default
|
\noun default
|
||||||
camera acting as the virtual representation of the mobile's rear-view camera.
|
camera acting as the virtual representation of the mobile's rear-view camera.
|
||||||
In both cases this virtual camera has a pose driver attached responsible
|
In both cases, this virtual camera has a pose driver attached responsible
|
||||||
for tracing the movements of the physical device in the virtual space and
|
for tracing the movements of the physical device in the virtual space and
|
||||||
a component responsible for displaying the view of the rear-facing camera
|
a component responsible for displaying the view of the rear-facing camera
|
||||||
as the background of the scene.
|
as the background of the scene.
|
||||||
This presents the core of the environment, additionally both libraries
|
This presents the core of the environment, additionally, both libraries
|
||||||
include separate components responsible for displaying discovered planes
|
include separate components responsible for displaying discovered planes
|
||||||
within the space, optionally with UI directing the user to aid in this
|
within the space, optionally with UI directing the user to aid in this
|
||||||
discovery.
|
discovery.
|
||||||
@ -6233,19 +6213,6 @@ name "sec:Frame-Rate-Throttling"
|
|||||||
\end_inset
|
\end_inset
|
||||||
|
|
||||||
|
|
||||||
\end_layout
|
|
||||||
|
|
||||||
\begin_layout Standard
|
|
||||||
\begin_inset Flex TODO Note (Margin)
|
|
||||||
status open
|
|
||||||
|
|
||||||
\begin_layout Plain Layout
|
|
||||||
diagrams
|
|
||||||
\end_layout
|
|
||||||
|
|
||||||
\end_inset
|
|
||||||
|
|
||||||
|
|
||||||
\end_layout
|
\end_layout
|
||||||
|
|
||||||
\begin_layout Standard
|
\begin_layout Standard
|
||||||
@ -6295,7 +6262,7 @@ Design Considerations
|
|||||||
|
|
||||||
\begin_layout Standard
|
\begin_layout Standard
|
||||||
When considering the final applications of these concepts in the context
|
When considering the final applications of these concepts in the context
|
||||||
of a multi-source project, the ability to have a step per source was included
|
of a multi-source project, the ability to have a step per-source was included
|
||||||
in order to take into account the different network conditions between
|
in order to take into account the different network conditions between
|
||||||
the server and each source's client(s).
|
the server and each source's client(s).
|
||||||
A single set of global FPS and delay requirements would still be provided
|
A single set of global FPS and delay requirements would still be provided
|
||||||
@ -6339,7 +6306,7 @@ The existing calculations of live statistics provided the global FPS and
|
|||||||
KinectServer
|
KinectServer
|
||||||
\noun default
|
\noun default
|
||||||
.
|
.
|
||||||
In order to collect these values on a source-by-source basis a dictionary
|
In order to collect these values on a source-by-source basis, a dictionary
|
||||||
of
|
of
|
||||||
\noun on
|
\noun on
|
||||||
SourceStat
|
SourceStat
|
||||||
@ -6365,14 +6332,14 @@ noprefix "false"
|
|||||||
|
|
||||||
, this iterative version was well suited to processing each collected latency
|
, this iterative version was well suited to processing each collected latency
|
||||||
value.
|
value.
|
||||||
Initially the values were found by collecting the latency of each frame
|
Initially, the values were found by collecting the latency of each frame
|
||||||
currently within the server's buffers.
|
currently within the server's buffers.
|
||||||
Theoretically this would have allowed the calculated average to be controlled
|
Theoretically, this would have allowed the calculated average to be controlled
|
||||||
by the speed with which frames are moved through the buffer.
|
by the speed with which frames are moved through the buffer.
|
||||||
However this was found to be ineffective as the buffers typically stay
|
However, this was found to be ineffective as the buffers typically stay
|
||||||
empty.
|
empty.
|
||||||
Instead a queue of fixed size was defined in order to retain the most recent
|
Instead, a queue of fixed size was defined in order to retain the most
|
||||||
values of latency received at the server.
|
recent values of latency received at the server.
|
||||||
This queue included calculations for both exponential and simple moving
|
This queue included calculations for both exponential and simple moving
|
||||||
averages of the contents.
|
averages of the contents.
|
||||||
\end_layout
|
\end_layout
|
||||||
@ -6480,7 +6447,7 @@ In order to function as a proof-of-concept for future investigations a simple
|
|||||||
\end_layout
|
\end_layout
|
||||||
|
|
||||||
\begin_layout Standard
|
\begin_layout Standard
|
||||||
In practice this meant that for each second a source's latency was higher
|
In practice, this meant that for each second a source's latency was higher
|
||||||
than the requirement, for example, the source's frame drop rate would be
|
than the requirement, for example, the source's frame drop rate would be
|
||||||
increased by 5%.
|
increased by 5%.
|
||||||
The implementation in code can be seen in appendix
|
The implementation in code can be seen in appendix
|
||||||
@ -6499,7 +6466,7 @@ noprefix "false"
|
|||||||
\begin_layout Standard
|
\begin_layout Standard
|
||||||
Future investigations could use a non-linear calculation in order to reflect
|
Future investigations could use a non-linear calculation in order to reflect
|
||||||
the distance from which operating values are from the requirements.
|
the distance from which operating values are from the requirements.
|
||||||
Theoretically this would allow the step to move rapidly in order to quickly
|
Theoretically, this would allow the step to move rapidly in order to quickly
|
||||||
approach the requirements before slowing and settling.
|
approach the requirements before slowing and settling.
|
||||||
\end_layout
|
\end_layout
|
||||||
|
|
||||||
@ -6530,9 +6497,68 @@ noprefix "false"
|
|||||||
Premise Validation
|
Premise Validation
|
||||||
\end_layout
|
\end_layout
|
||||||
|
|
||||||
|
\begin_layout Standard
|
||||||
|
\begin_inset Float figure
|
||||||
|
wide false
|
||||||
|
sideways false
|
||||||
|
status open
|
||||||
|
|
||||||
|
\begin_layout Plain Layout
|
||||||
|
\noindent
|
||||||
|
\align center
|
||||||
|
\begin_inset Graphics
|
||||||
|
filename ../media/TestDev.png
|
||||||
|
lyxscale 30
|
||||||
|
width 45col%
|
||||||
|
|
||||||
|
\end_inset
|
||||||
|
|
||||||
|
|
||||||
|
\end_layout
|
||||||
|
|
||||||
|
\begin_layout Plain Layout
|
||||||
|
\begin_inset Caption Standard
|
||||||
|
|
||||||
|
\begin_layout Plain Layout
|
||||||
|
Client-server relationship in a development environment located on a single
|
||||||
|
machine
|
||||||
|
\begin_inset CommandInset label
|
||||||
|
LatexCommand label
|
||||||
|
name "fig:TestDev"
|
||||||
|
|
||||||
|
\end_inset
|
||||||
|
|
||||||
|
|
||||||
|
\end_layout
|
||||||
|
|
||||||
|
\end_inset
|
||||||
|
|
||||||
|
|
||||||
|
\end_layout
|
||||||
|
|
||||||
|
\begin_layout Plain Layout
|
||||||
|
|
||||||
|
\end_layout
|
||||||
|
|
||||||
|
\end_inset
|
||||||
|
|
||||||
|
|
||||||
|
\end_layout
|
||||||
|
|
||||||
\begin_layout Standard
|
\begin_layout Standard
|
||||||
Development was conducted with the client and server located on the same
|
Development was conducted with the client and server located on the same
|
||||||
machine reducing the latency to just that of processing the frames themselves.
|
machine reducing the latency to just that of processing the frames themselves,
|
||||||
|
see figure
|
||||||
|
\begin_inset CommandInset ref
|
||||||
|
LatexCommand ref
|
||||||
|
reference "fig:TestDev"
|
||||||
|
plural "false"
|
||||||
|
caps "false"
|
||||||
|
noprefix "false"
|
||||||
|
|
||||||
|
\end_inset
|
||||||
|
|
||||||
|
.
|
||||||
In order to validate the premise of the hypothesis, investigations were
|
In order to validate the premise of the hypothesis, investigations were
|
||||||
made on two machines across a domestic local area network.
|
made on two machines across a domestic local area network.
|
||||||
\end_layout
|
\end_layout
|
||||||
@ -6619,13 +6645,57 @@ The small scale of the controlled LAN environment does not inherently incur
|
|||||||
As the full-scene configuration requires around 275Mbps, artificial latency
|
As the full-scene configuration requires around 275Mbps, artificial latency
|
||||||
was introduced as frames took longer to deliver and were produced faster
|
was introduced as frames took longer to deliver and were produced faster
|
||||||
than they could be transmitted to the server.
|
than they could be transmitted to the server.
|
||||||
|
A diagram of this testing layout can be seen in figure .
|
||||||
|
\end_layout
|
||||||
|
|
||||||
|
\begin_layout Standard
|
||||||
|
\begin_inset Float figure
|
||||||
|
wide false
|
||||||
|
sideways false
|
||||||
|
status open
|
||||||
|
|
||||||
|
\begin_layout Plain Layout
|
||||||
|
\noindent
|
||||||
|
\align center
|
||||||
|
\begin_inset Graphics
|
||||||
|
filename ../media/TestLAN.png
|
||||||
|
lyxscale 30
|
||||||
|
width 50col%
|
||||||
|
|
||||||
|
\end_inset
|
||||||
|
|
||||||
|
|
||||||
|
\end_layout
|
||||||
|
|
||||||
|
\begin_layout Plain Layout
|
||||||
|
\begin_inset Caption Standard
|
||||||
|
|
||||||
|
\begin_layout Plain Layout
|
||||||
|
Client-server relationship during initial testing in a LAN environment
|
||||||
|
\begin_inset CommandInset label
|
||||||
|
LatexCommand label
|
||||||
|
name "fig:TestLAN"
|
||||||
|
|
||||||
|
\end_inset
|
||||||
|
|
||||||
|
|
||||||
|
\end_layout
|
||||||
|
|
||||||
|
\end_inset
|
||||||
|
|
||||||
|
|
||||||
|
\end_layout
|
||||||
|
|
||||||
|
\end_inset
|
||||||
|
|
||||||
|
|
||||||
\end_layout
|
\end_layout
|
||||||
|
|
||||||
\begin_layout Standard
|
\begin_layout Standard
|
||||||
The step was specified and varied manually initially in order to visualise
|
The step was specified and varied manually initially in order to visualise
|
||||||
the response in delay and FPS.
|
the response in delay and FPS.
|
||||||
Following implementation of a dynamic step, the response was again visualised
|
Following the implementation of a dynamic step, the response was again
|
||||||
using figure-only capture for realistic latency values.
|
visualised using figure-only capture for realistic latency values.
|
||||||
\end_layout
|
\end_layout
|
||||||
|
|
||||||
\begin_layout Subsubsection
|
\begin_layout Subsubsection
|
||||||
@ -6644,6 +6714,17 @@ Microsoft
|
|||||||
Azure
|
Azure
|
||||||
\noun default
|
\noun default
|
||||||
cloud computing environment.
|
cloud computing environment.
|
||||||
|
This final testing scenario can be seen in figure
|
||||||
|
\begin_inset CommandInset ref
|
||||||
|
LatexCommand ref
|
||||||
|
reference "fig:TestCloud"
|
||||||
|
plural "false"
|
||||||
|
caps "false"
|
||||||
|
noprefix "false"
|
||||||
|
|
||||||
|
\end_inset
|
||||||
|
|
||||||
|
.
|
||||||
A
|
A
|
||||||
\emph on
|
\emph on
|
||||||
Standard F2s V2
|
Standard F2s V2
|
||||||
@ -6670,6 +6751,49 @@ When operating over the open internet the domestic internet connection became
|
|||||||
As such figure-only capture was used in order to limit the required bandwidth.
|
As such figure-only capture was used in order to limit the required bandwidth.
|
||||||
\end_layout
|
\end_layout
|
||||||
|
|
||||||
|
\begin_layout Standard
|
||||||
|
\begin_inset Float figure
|
||||||
|
wide false
|
||||||
|
sideways false
|
||||||
|
status open
|
||||||
|
|
||||||
|
\begin_layout Plain Layout
|
||||||
|
\noindent
|
||||||
|
\align center
|
||||||
|
\begin_inset Graphics
|
||||||
|
filename ../media/TestCloud.png
|
||||||
|
lyxscale 30
|
||||||
|
width 50col%
|
||||||
|
|
||||||
|
\end_inset
|
||||||
|
|
||||||
|
|
||||||
|
\end_layout
|
||||||
|
|
||||||
|
\begin_layout Plain Layout
|
||||||
|
\begin_inset Caption Standard
|
||||||
|
|
||||||
|
\begin_layout Plain Layout
|
||||||
|
Client-server relationship during further testing in a cloud-based environment
|
||||||
|
\begin_inset CommandInset label
|
||||||
|
LatexCommand label
|
||||||
|
name "fig:TestCloud"
|
||||||
|
|
||||||
|
\end_inset
|
||||||
|
|
||||||
|
|
||||||
|
\end_layout
|
||||||
|
|
||||||
|
\end_inset
|
||||||
|
|
||||||
|
|
||||||
|
\end_layout
|
||||||
|
|
||||||
|
\end_inset
|
||||||
|
|
||||||
|
|
||||||
|
\end_layout
|
||||||
|
|
||||||
\begin_layout Standard
|
\begin_layout Standard
|
||||||
\begin_inset Newpage newpage
|
\begin_inset Newpage newpage
|
||||||
\end_inset
|
\end_inset
|
||||||
@ -6768,7 +6892,7 @@ name "fig:Latency-and-FPS1"
|
|||||||
\end_layout
|
\end_layout
|
||||||
|
|
||||||
\begin_layout Standard
|
\begin_layout Standard
|
||||||
Generally the lowering of the drop rate induces the latency to begin increasing
|
Generally, the lowering of the drop rate induces the latency to begin increasing
|
||||||
at a fairly linear rate.
|
at a fairly linear rate.
|
||||||
This can be seen in figure
|
This can be seen in figure
|
||||||
\begin_inset CommandInset ref
|
\begin_inset CommandInset ref
|
||||||
@ -6806,7 +6930,7 @@ noprefix "false"
|
|||||||
\begin_layout Standard
|
\begin_layout Standard
|
||||||
Looking to the FPS chart, the frame rate drops in response to an increase
|
Looking to the FPS chart, the frame rate drops in response to an increase
|
||||||
in frame drop rate and vice versa.
|
in frame drop rate and vice versa.
|
||||||
Similarly to latency a lag can be seen when changes in drop rate are received.
|
Similarly to latency, a lag can be seen when changes in drop rate are received.
|
||||||
Worth noticing is that as the step drops, the FPS increases before the
|
Worth noticing is that as the step drops, the FPS increases before the
|
||||||
latency begins increasing (
|
latency begins increasing (
|
||||||
\begin_inset Formula $t=70$
|
\begin_inset Formula $t=70$
|
||||||
@ -6844,8 +6968,8 @@ noprefix "false"
|
|||||||
\end_inset
|
\end_inset
|
||||||
|
|
||||||
.
|
.
|
||||||
It can be seen that as the average latency increases above the prescribed
|
It can be seen that as the average latency (blue) increased above the prescribe
|
||||||
latency requirement an increase in dynamic step was induced.
|
d latency requirement (green) an increase in dynamic step (red) was induced.
|
||||||
As the latency fell past the requirement, the step also fell.
|
As the latency fell past the requirement, the step also fell.
|
||||||
\end_layout
|
\end_layout
|
||||||
|
|
||||||
@ -7106,7 +7230,7 @@ noprefix "false"
|
|||||||
|
|
||||||
is presented to highlight the instability encountered in the cloud-based
|
is presented to highlight the instability encountered in the cloud-based
|
||||||
environment.
|
environment.
|
||||||
Similarly to the local environment an initial spike in latency is seen
|
Similarly to the local environment, an initial spike in latency is seen
|
||||||
before reducing to a consistent value.
|
before reducing to a consistent value.
|
||||||
When the frame drop rate is dropped from 0.8 to 0.6, however, the connection
|
When the frame drop rate is dropped from 0.8 to 0.6, however, the connection
|
||||||
becomes immediately unstable with bandwidth dropping to 0 Mbps as the connectio
|
becomes immediately unstable with bandwidth dropping to 0 Mbps as the connectio
|
||||||
@ -7203,7 +7327,7 @@ LAN Premise Validation
|
|||||||
The lag in latency response could be attributed to a couple of influencing
|
The lag in latency response could be attributed to a couple of influencing
|
||||||
factors.
|
factors.
|
||||||
The nature of a moving average partially representing previous values when
|
The nature of a moving average partially representing previous values when
|
||||||
iterating adds a
|
iterating adds
|
||||||
\emph on
|
\emph on
|
||||||
friction
|
friction
|
||||||
\emph default
|
\emph default
|
||||||
@ -7217,7 +7341,7 @@ The initial spike in latency could be a result of the delay in drop rate
|
|||||||
Additionally, the drop rate was employed at the client prior to queueing
|
Additionally, the drop rate was employed at the client prior to queueing
|
||||||
frames in the transmission buffer.
|
frames in the transmission buffer.
|
||||||
Frames are then delivered to the server decoupled from this rate.
|
Frames are then delivered to the server decoupled from this rate.
|
||||||
As a result when a new drop rate is received the buffer is filled at this
|
As a result, when a new drop rate is received the buffer is filled at this
|
||||||
rate going forward however the existing population queued at a different
|
rate going forward however the existing population queued at a different
|
||||||
rate must still be transmitted.
|
rate must still be transmitted.
|
||||||
\end_layout
|
\end_layout
|
||||||
@ -7225,11 +7349,10 @@ The initial spike in latency could be a result of the delay in drop rate
|
|||||||
\begin_layout Standard
|
\begin_layout Standard
|
||||||
Following the implementation of a dynamic step, the latency was no longer
|
Following the implementation of a dynamic step, the latency was no longer
|
||||||
unbounded and instead oscillated as the step increased and decreased.
|
unbounded and instead oscillated as the step increased and decreased.
|
||||||
As a result the system can be seen to dynamically drop frames in order
|
As a result, the system can be seen to dynamically drop frames in order
|
||||||
to maintain a constant latency in adverse network conditions.
|
to maintain a constant latency in adverse network conditions.
|
||||||
This allows the system to keep up when when the amount of frames being
|
This allows the system to keep up when the amount of frames being produced
|
||||||
produced by the client is too high to be transmitted in the same period
|
by the client is too high to be transmitted in the same period of time.
|
||||||
of time.
|
|
||||||
\end_layout
|
\end_layout
|
||||||
|
|
||||||
\begin_layout Standard
|
\begin_layout Standard
|
||||||
@ -7245,11 +7368,10 @@ The performance of this system, however, should be considered.
|
|||||||
|
|
||||||
\begin_layout Standard
|
\begin_layout Standard
|
||||||
One method to control this would be to have the step move in a non-linear
|
One method to control this would be to have the step move in a non-linear
|
||||||
fashion, increasing at a higher rate the further away the live value is
|
fashion, changing at a rate proportional to the difference between the
|
||||||
from the requirement and dropping faster as it falls below the required
|
live value and the requirement.
|
||||||
value.
|
|
||||||
This could allow the latency to oscillate closer to the requirement with
|
This could allow the latency to oscillate closer to the requirement with
|
||||||
the goal being for it to settle at that value.
|
the goal being for it to settle at this value.
|
||||||
\end_layout
|
\end_layout
|
||||||
|
|
||||||
\begin_layout Subsubsection
|
\begin_layout Subsubsection
|
||||||
@ -7268,7 +7390,7 @@ With the employed method of controlling transmission latency, representative
|
|||||||
\begin_layout Standard
|
\begin_layout Standard
|
||||||
The need to deliver a step from the server to each client created an initial
|
The need to deliver a step from the server to each client created an initial
|
||||||
spike in latency in both environments but was more pronounced in the cloud.
|
spike in latency in both environments but was more pronounced in the cloud.
|
||||||
Additionally there were issues with the step not being delivered to the
|
Additionally, there were issues with the step not being delivered to the
|
||||||
client at all, as this is included in the header of each request it was
|
client at all, as this is included in the header of each request it was
|
||||||
concluded that requests were either being lost or queued at the server
|
concluded that requests were either being lost or queued at the server
|
||||||
unable to be transmitted over the socket busy receiving frames.
|
unable to be transmitted over the socket busy receiving frames.
|
||||||
@ -7337,8 +7459,8 @@ active
|
|||||||
\noun on
|
\noun on
|
||||||
Kinect
|
Kinect
|
||||||
\noun default
|
\noun default
|
||||||
sensors a method to tightly match different source frames within the processing
|
sensors, a method to tightly match different source frames within the processin
|
||||||
pipeline based on timestamp could be implemented and tested.
|
g pipeline based on timestamp could be implemented and tested.
|
||||||
\end_layout
|
\end_layout
|
||||||
|
|
||||||
\begin_layout Standard
|
\begin_layout Standard
|
||||||
@ -7376,8 +7498,8 @@ ARFoundation
|
|||||||
Metal
|
Metal
|
||||||
\noun default
|
\noun default
|
||||||
graphics library, limiting functionality to just that of the network layer.
|
graphics library, limiting functionality to just that of the network layer.
|
||||||
With a compatible shader the cross-platform migration would be feature-complete
|
With a compatible shader, the cross-platform migration would be feature-complet
|
||||||
, extending the possible install base of the application while also providing
|
e, extending the possible install base of the application while also providing
|
||||||
access to the currently wider feature set of iOS's
|
access to the currently wider feature set of iOS's
|
||||||
\noun on
|
\noun on
|
||||||
ARKit
|
ARKit
|
||||||
@ -7433,6 +7555,8 @@ frame types
|
|||||||
will change significantly each frame and as such calculating and transmitting
|
will change significantly each frame and as such calculating and transmitting
|
||||||
only the changes since the last could significantly reduce the required
|
only the changes since the last could significantly reduce the required
|
||||||
bandwidth.
|
bandwidth.
|
||||||
|
The challenge here would be establishing temporal coherence between successive
|
||||||
|
frames to identify these differences, a harder task in 3D.
|
||||||
\end_layout
|
\end_layout
|
||||||
|
|
||||||
\begin_layout Standard
|
\begin_layout Standard
|
||||||
@ -7454,7 +7578,7 @@ name "sec:Summary"
|
|||||||
\end_layout
|
\end_layout
|
||||||
|
|
||||||
\begin_layout Standard
|
\begin_layout Standard
|
||||||
Within this piece the process of extending the
|
Within this piece, the process of extending the
|
||||||
\noun on
|
\noun on
|
||||||
LiveScan3D
|
LiveScan3D
|
||||||
\noun default
|
\noun default
|
||||||
@ -7480,11 +7604,11 @@ The
|
|||||||
\noun on
|
\noun on
|
||||||
LiveScan3D
|
LiveScan3D
|
||||||
\noun default
|
\noun default
|
||||||
suite is described, both in it's initial state and following developments
|
suite is described, both in its initial state and following developments
|
||||||
made to the network behaviour to include a sub-system of buffers.
|
made to the network behaviour to include a sub-system of buffers.
|
||||||
Both accompanying AR applications for the suite are presented and finally
|
Both accompanying AR applications for the suite are presented and finally
|
||||||
the suite as a whole is evaluated with it's strengths and limitations highlight
|
the suite as a whole is evaluated with its strengths and limitations highlighte
|
||||||
ed.
|
d.
|
||||||
\end_layout
|
\end_layout
|
||||||
|
|
||||||
\begin_layout Standard
|
\begin_layout Standard
|
||||||
@ -7616,6 +7740,30 @@ options "bibtotoc"
|
|||||||
|
|
||||||
\begin_layout Section
|
\begin_layout Section
|
||||||
\start_of_appendix
|
\start_of_appendix
|
||||||
|
Project Gantt Chart
|
||||||
|
\begin_inset CommandInset label
|
||||||
|
LatexCommand label
|
||||||
|
name "sec:Gantt-Chart"
|
||||||
|
|
||||||
|
\end_inset
|
||||||
|
|
||||||
|
|
||||||
|
\end_layout
|
||||||
|
|
||||||
|
\begin_layout Standard
|
||||||
|
\noindent
|
||||||
|
\align center
|
||||||
|
\begin_inset Graphics
|
||||||
|
filename ../media/GanttChart.png
|
||||||
|
lyxscale 30
|
||||||
|
width 100col%
|
||||||
|
|
||||||
|
\end_inset
|
||||||
|
|
||||||
|
|
||||||
|
\end_layout
|
||||||
|
|
||||||
|
\begin_layout Section
|
||||||
Server UI Additions
|
Server UI Additions
|
||||||
\begin_inset CommandInset label
|
\begin_inset CommandInset label
|
||||||
LatexCommand label
|
LatexCommand label
|
||||||
|
BIN
media/GanttChart.png
Normal file
BIN
media/GanttChart.png
Normal file
Binary file not shown.
After Width: | Height: | Size: 127 KiB |
BIN
media/TestCloud.png
Normal file
BIN
media/TestCloud.png
Normal file
Binary file not shown.
After Width: | Height: | Size: 96 KiB |
BIN
media/TestDev.png
Normal file
BIN
media/TestDev.png
Normal file
Binary file not shown.
After Width: | Height: | Size: 42 KiB |
BIN
media/TestLAN.png
Normal file
BIN
media/TestLAN.png
Normal file
Binary file not shown.
After Width: | Height: | Size: 57 KiB |
BIN
resources/TestScenarios.odg
Normal file
BIN
resources/TestScenarios.odg
Normal file
Binary file not shown.
Loading…
Reference in New Issue
Block a user