Dmitriy Gizlyk / 프로필
- 정보
10+ 년도
경험
|
0
제품
|
0
데몬 버전
|
134
작업
|
0
거래 신호
|
0
구독자
|
![Neural networks made easy (Part 69): Density-based support constraint for the behavioral policy (SPOT)](https://c.mql5.com/2/63/midjourney_image_13954_55_495__1-logo__1.png)
In offline learning, we use a fixed dataset, which limits the coverage of environmental diversity. During the learning process, our Agent can generate actions beyond this dataset. If there is no feedback from the environment, how can we be sure that the assessments of such actions are correct? Maintaining the Agent's policy within the training dataset becomes an important aspect to ensure the reliability of training. This is what we will talk about in this article.
![JimReaper JimReaper](https://c.mql5.com/avatar/2023/9/650d3819-17e6.jpg)
![Neural networks made easy (Part 68): Offline Preference-guided Policy Optimization](https://c.mql5.com/2/62/midjourney_image_13912_49_444__1-logo.png)
Since the first articles devoted to reinforcement learning, we have in one way or another touched upon 2 problems: exploring the environment and determining the reward function. Recent articles have been devoted to the problem of exploration in offline learning. In this article, I would like to introduce you to an algorithm whose authors completely eliminated the reward function.
![Neural networks made easy (Part 67): Using past experience to solve new tasks](https://c.mql5.com/2/62/Neural_networks_made_easy_Part_67__LOGO.png)
In this article, we continue discussing methods for collecting data into a training set. Obviously, the learning process requires constant interaction with the environment. However, situations can be different.
![JimReaper JimReaper](https://c.mql5.com/avatar/2023/9/650d3819-17e6.jpg)
![Neural networks made easy (Part 66): Exploration problems in offline learning](https://c.mql5.com/2/61/Neural_networks_are_easy_Part_66_LOGO.png)
Models are trained offline using data from a prepared training dataset. While providing certain advantages, its negative side is that information about the environment is greatly compressed to the size of the training dataset. Which, in turn, limits the possibilities of exploration. In this article, we will consider a method that enables the filling of a training dataset with the most diverse data possible.
![JimReaper JimReaper](https://c.mql5.com/avatar/2023/9/650d3819-17e6.jpg)
![Neural networks made easy (Part 65): Distance Weighted Supervised Learning (DWSL)](https://c.mql5.com/2/61/Neural_Networks_Made_Easy_lPart_65q_DWSL_LOGO.png)
In this article, we will get acquainted with an interesting algorithm that is built at the intersection of supervised and reinforcement learning methods.
![Neural networks made easy (Part 64): ConserWeightive Behavioral Cloning (CWBC) method](https://c.mql5.com/2/60/Neural_networks_made_easy_rPart_64s_CWBC__Logo.png)
As a result of tests performed in previous articles, we came to the conclusion that the optimality of the trained strategy largely depends on the training set used. In this article, we will get acquainted with a fairly simple yet effective method for selecting trajectories to train models.
![Neural networks made easy (Part 63): Unsupervised Pretraining for Decision Transformer (PDT)](https://c.mql5.com/2/60/Neural_networks_are_easy_wPart_636_Logo.png)
We continue to discuss the family of Decision Transformer methods. From previous article, we have already noticed that training the transformer underlying the architecture of these methods is a rather complex task and requires a large labeled dataset for training. In this article we will look at an algorithm for using unlabeled trajectories for preliminary model training.
![Neural networks made easy (Part 62): Using Decision Transformer in hierarchical models](https://c.mql5.com/2/59/Neural_networks_are_easy_0Part_62s_logo.png)
In recent articles, we have seen several options for using the Decision Transformer method. The method allows analyzing not only the current state, but also the trajectory of previous states and actions performed in them. In this article, we will focus on using this method in hierarchical models.
![Neural networks made easy (Part 61): Optimism issue in offline reinforcement learning](https://c.mql5.com/2/59/NN_easy_61_Logo__V4_.png)
During the offline learning, we optimize the Agent's policy based on the training sample data. The resulting strategy gives the Agent confidence in its actions. However, such optimism is not always justified and can cause increased risks during the model operation. Today we will look at one of the methods to reduce these risks.
![Neural networks made easy (Part 60): Online Decision Transformer (ODT)](https://c.mql5.com/2/59/Online_Decision_Transformer_logo_up.png)
The last two articles were devoted to the Decision Transformer method, which models action sequences in the context of an autoregressive model of desired rewards. In this article, we will look at another optimization algorithm for this method.
![Neural networks are easy (Part 59): Dichotomy of Control (DoC)](https://c.mql5.com/2/58/logo__1.png)
In the previous article, we got acquainted with the Decision Transformer. But the complex stochastic environment of the foreign exchange market did not allow us to fully implement the potential of the presented method. In this article, I will introduce an algorithm that is aimed at improving the performance of algorithms in stochastic environments.
![Neural networks made easy (Part 58): Decision Transformer (DT)](https://c.mql5.com/2/58/decision-transformer-avatar.png)
We continue to explore reinforcement learning methods. In this article, I will focus on a slightly different algorithm that considers the Agent’s policy in the paradigm of constructing a sequence of actions.
![Yao Wei Lai Yao Wei Lai](https://c.mql5.com/avatar/avatar_na2_micro.png)
![Neural networks made easy (Part 57): Stochastic Marginal Actor-Critic (SMAC)](https://c.mql5.com/2/58/stochastic_marginal_actor_critic_avatar.png)
Here I will consider the fairly new Stochastic Marginal Actor-Critic (SMAC) algorithm, which allows building latent variable policies within the framework of entropy maximization.
![Neural networks made easy (Part 56): Using nuclear norm to drive research](https://c.mql5.com/2/57/nuclear_norm_utilization_avatar.png)
The study of the environment in reinforcement learning is a pressing problem. We have already looked at some approaches previously. In this article, we will have a look at yet another method based on maximizing the nuclear norm. It allows agents to identify environmental states with a high degree of novelty and diversity.
![Neural networks made easy (Part 55): Contrastive intrinsic control (CIC)](https://c.mql5.com/2/57/cic-055-avatar.png)
Contrastive training is an unsupervised method of training representation. Its goal is to train a model to highlight similarities and differences in data sets. In this article, we will talk about using contrastive training approaches to explore different Actor skills.
![Neural networks made easy (Part 54): Using random encoder for efficient research (RE3)](https://c.mql5.com/2/57/random_encoder_for_efficient_exploration_054_avatar.png)
Whenever we consider reinforcement learning methods, we are faced with the issue of efficiently exploring the environment. Solving this issue often leads to complication of the algorithm and training of additional models. In this article, we will look at an alternative approach to solving this problem.
![Neural networks made easy (Part 53): Reward decomposition](https://c.mql5.com/2/57/decomposition_of_remuneration_053_avatar.png)
We have already talked more than once about the importance of correctly selecting the reward function, which we use to stimulate the desired behavior of the Agent by adding rewards or penalties for individual actions. But the question remains open about the decryption of our signals by the Agent. In this article, we will talk about reward decomposition in terms of transmitting individual signals to the trained Agent.
![Neural networks made easy (Part 52): Research with optimism and distribution correction](https://c.mql5.com/2/57/optimistic-actor-critic-avatar.png)
As the model is trained based on the experience reproduction buffer, the current Actor policy moves further and further away from the stored examples, which reduces the efficiency of training the model as a whole. In this article, we will look at the algorithm of improving the efficiency of using samples in reinforcement learning algorithms.
![Neural networks made easy (Part 51): Behavior-Guided Actor-Critic (BAC)](https://c.mql5.com/2/57/behavior_driven_actor_critic_avatar.png)
The last two articles considered the Soft Actor-Critic algorithm, which incorporates entropy regularization into the reward function. This approach balances environmental exploration and model exploitation, but it is only applicable to stochastic models. The current article proposes an alternative approach that is applicable to both stochastic and deterministic models.
![Neural networks made easy (Part 50): Soft Actor-Critic (model optimization)](https://c.mql5.com/2/57/NN_50_Soft_Actor-Critic_Avatar.png)
In the previous article, we implemented the Soft Actor-Critic algorithm, but were unable to train a profitable model. Here we will optimize the previously created model to obtain the desired results.